==================== Test output for //tensorflow/tsl/profiler/rpc/client:profiler_client_test: [==========] Running 5 tests from 1 test suite. [----------] Global test environment set-up. [----------] 5 tests from RemoteProfilerSession [ RUN ] RemoteProfilerSession.Simple 2023-07-23 21:56:38.123515: I tensorflow/tsl/profiler/rpc/profiler_server.cc:46] Profiler server listening on [::]:33782 selected port:33782 2023-07-23 21:56:38.123713: I ./tensorflow/tsl/profiler/rpc/client/profiler_client_test_util.h:62] Started localhost:33782 at 2023-07-23T21:56:38.123670428+00:00 2023-07-23 21:56:38.123736: I ./tensorflow/tsl/profiler/rpc/client/profiler_client_test_util.h:63] Duration: 10ms 2023-07-23 21:56:38.123897: I tensorflow/tsl/profiler/rpc/client/profiler_client.cc:122] Asynchronous gRPC Profile() to localhost:33782 2023-07-23 21:56:38.124036: I tensorflow/tsl/profiler/rpc/client/profiler_client.cc:140] Waiting for completion. 2023-07-23 21:56:38.176458: I tensorflow/tsl/profiler/lib/profiler_session.cc:104] Profiler session initializing. 2023-07-23 21:56:38.176516: I tensorflow/tsl/profiler/lib/profiler_session.cc:119] Profiler session started. 2023-07-23 21:56:38.196229: I tensorflow/tsl/profiler/lib/profiler_session.cc:70] Profiler session collecting data. 2023-07-23 21:56:38.196594: I tensorflow/tsl/profiler/rpc/client/save_profile.cc:144] Collecting XSpace to repository: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/be21e2e08a187daa4953c308b5ee489d/test_session/localhost_33782.xplane.pb 2023-07-23 21:56:38.196823: I tensorflow/tsl/profiler/lib/profiler_session.cc:131] Profiler session tear down. 2023-07-23 21:56:38.198108: I tensorflow/tsl/profiler/rpc/profiler_server.cc:55] Profiler server was shut down [ OK ] RemoteProfilerSession.Simple (77 ms) [ RUN ] RemoteProfilerSession.WaitNotCalled 2023-07-23 21:56:38.198741: I tensorflow/tsl/profiler/rpc/profiler_server.cc:46] Profiler server listening on [::]:41455 selected port:41455 2023-07-23 21:56:38.198802: I ./tensorflow/tsl/profiler/rpc/client/profiler_client_test_util.h:62] Started localhost:41455 at 2023-07-23T21:56:38.198782639+00:00 2023-07-23 21:56:38.198818: I ./tensorflow/tsl/profiler/rpc/client/profiler_client_test_util.h:63] Duration: 10ms 2023-07-23 21:56:38.198962: I tensorflow/tsl/profiler/rpc/client/profiler_client.cc:122] Asynchronous gRPC Profile() to localhost:41455 2023-07-23 21:56:38.199113: I tensorflow/tsl/profiler/rpc/client/profiler_client.cc:140] Waiting for completion. 2023-07-23 21:56:38.201042: I tensorflow/tsl/profiler/lib/profiler_session.cc:104] Profiler session initializing. 2023-07-23 21:56:38.201079: I tensorflow/tsl/profiler/lib/profiler_session.cc:119] Profiler session started. 2023-07-23 21:56:38.211402: I tensorflow/tsl/profiler/lib/profiler_session.cc:70] Profiler session collecting data. 2023-07-23 21:56:38.211598: I tensorflow/tsl/profiler/rpc/client/save_profile.cc:144] Collecting XSpace to repository: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/be21e2e08a187daa4953c308b5ee489d/test_session/localhost_41455.xplane.pb 2023-07-23 21:56:38.211897: I tensorflow/tsl/profiler/lib/profiler_session.cc:131] Profiler session tear down. 2023-07-23 21:56:38.234638: I tensorflow/tsl/profiler/rpc/profiler_server.cc:55] Profiler server was shut down [ OK ] RemoteProfilerSession.WaitNotCalled (36 ms) [ RUN ] RemoteProfilerSession.Timeout 2023-07-23 21:56:38.236050: I tensorflow/tsl/profiler/rpc/profiler_server.cc:46] Profiler server listening on [::]:40366 selected port:40366 2023-07-23 21:56:38.236149: I ./tensorflow/tsl/profiler/rpc/client/profiler_client_test_util.h:62] Started localhost:40366 at 2023-07-23T21:56:38.236105715+00:00 2023-07-23 21:56:38.236183: I ./tensorflow/tsl/profiler/rpc/client/profiler_client_test_util.h:63] Duration: 10ms 2023-07-23 21:56:38.236325: I tensorflow/tsl/profiler/rpc/client/profiler_client.cc:122] Asynchronous gRPC Profile() to localhost:40366 2023-07-23 21:56:38.236480: I tensorflow/tsl/profiler/rpc/client/profiler_client.cc:140] Waiting for completion. 2023-07-23 21:56:38.237230: W tensorflow/tsl/profiler/rpc/client/profiler_client.cc:161] DEADLINE_EXCEEDED: Deadline Exceeded 2023-07-23 21:56:38.238464: I tensorflow/tsl/profiler/rpc/profiler_server.cc:55] Profiler server was shut down [ OK ] RemoteProfilerSession.Timeout (3 ms) [ RUN ] RemoteProfilerSession.LongDeadline 2023-07-23 21:56:38.257900: I tensorflow/tsl/profiler/rpc/profiler_server.cc:46] Profiler server listening on [::]:39004 selected port:39004 2023-07-23 21:56:38.257996: I ./tensorflow/tsl/profiler/rpc/client/profiler_client_test_util.h:62] Started localhost:39004 at 2023-07-23T21:56:38.257973951+00:00 2023-07-23 21:56:38.258016: I ./tensorflow/tsl/profiler/rpc/client/profiler_client_test_util.h:63] Duration: 10ms 2023-07-23 21:56:38.258159: I tensorflow/tsl/profiler/rpc/client/profiler_client.cc:122] Asynchronous gRPC Profile() to localhost:39004 2023-07-23 21:56:38.258309: I tensorflow/tsl/profiler/rpc/client/profiler_client.cc:140] Waiting for completion. 2023-07-23 21:56:38.259642: I tensorflow/tsl/profiler/lib/profiler_session.cc:104] Profiler session initializing. 2023-07-23 21:56:38.259674: I tensorflow/tsl/profiler/lib/profiler_session.cc:119] Profiler session started. 2023-07-23 21:56:38.270278: I tensorflow/tsl/profiler/lib/profiler_session.cc:70] Profiler session collecting data. 2023-07-23 21:56:38.270464: I tensorflow/tsl/profiler/rpc/client/save_profile.cc:144] Collecting XSpace to repository: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/be21e2e08a187daa4953c308b5ee489d/test_session/localhost_39004.xplane.pb 2023-07-23 21:56:38.270728: I tensorflow/tsl/profiler/lib/profiler_session.cc:131] Profiler session tear down. 2023-07-23 21:56:38.271987: I tensorflow/tsl/profiler/rpc/profiler_server.cc:55] Profiler server was shut down [ OK ] RemoteProfilerSession.LongDeadline (33 ms) [ RUN ] RemoteProfilerSession.LongDuration 2023-07-23 21:56:38.273228: I tensorflow/tsl/profiler/rpc/profiler_server.cc:46] Profiler server listening on [::]:59966 selected port:59966 2023-07-23 21:56:38.273292: I ./tensorflow/tsl/profiler/rpc/client/profiler_client_test_util.h:62] Started localhost:59966 at 2023-07-23T21:56:38.273271839+00:00 2023-07-23 21:56:38.273310: I ./tensorflow/tsl/profiler/rpc/client/profiler_client_test_util.h:63] Duration: 3s 2023-07-23 21:56:38.273456: I tensorflow/tsl/profiler/rpc/client/profiler_client.cc:122] Asynchronous gRPC Profile() to localhost:59966 2023-07-23 21:56:38.273599: I tensorflow/tsl/profiler/rpc/client/profiler_client.cc:140] Waiting for completion. 2023-07-23 21:56:38.274859: I tensorflow/tsl/profiler/lib/profiler_session.cc:104] Profiler session initializing. 2023-07-23 21:56:38.274888: I tensorflow/tsl/profiler/lib/profiler_session.cc:119] Profiler session started. 2023-07-23 21:56:41.275189: I tensorflow/tsl/profiler/lib/profiler_session.cc:70] Profiler session collecting data. 2023-07-23 21:56:42.419375: I tensorflow/tsl/profiler/rpc/client/save_profile.cc:144] Collecting XSpace to repository: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/be21e2e08a187daa4953c308b5ee489d/test_session/localhost_59966.xplane.pb 2023-07-23 21:56:42.419867: W tensorflow/tsl/profiler/rpc/client/profiler_client.cc:161] DEADLINE_EXCEEDED: Deadline Exceeded 2023-07-23 21:56:42.419919: I tensorflow/tsl/profiler/lib/profiler_session.cc:131] Profiler session tear down. tensorflow/tsl/profiler/rpc/client/profiler_client_test.cc:142: Failure Value of: status.ok() Actual: false Expected: true 2023-07-23 21:56:42.420624: I tensorflow/tsl/profiler/rpc/profiler_server.cc:55] Profiler server was shut down [ FAILED ] RemoteProfilerSession.LongDuration (4148 ms) [----------] 5 tests from RemoteProfilerSession (4299 ms total) [----------] Global test environment tear-down [==========] 5 tests from 1 test suite ran. (4299 ms total) [ PASSED ] 4 tests. [ FAILED ] 1 test, listed below: [ FAILED ] RemoteProfilerSession.LongDuration 1 FAILED TEST ================================================================================ ==================== Test output for //tensorflow/python/data/experimental/kernel_tests/service:local_workers_test (shard 1 of 24): Running tests under Python 3.10.9: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/python_aarch64-unknown-linux-gnu/bin/python3 [ RUN ] LocalTaskGarbageCollectTest.testMultipleEpochsSharedJob_test_mode_eager_tfapiversion_1_numremoteworkers_0 [ SKIPPED ] LocalTaskGarbageCollectTest.testMultipleEpochsSharedJob_test_mode_eager_tfapiversion_1_numremoteworkers_0 [ RUN ] LocalTaskGarbageCollectTest.testReadFromDeletedTask_test_mode_eager_tfapiversion_1_numremoteworkers_0 [ SKIPPED ] LocalTaskGarbageCollectTest.testReadFromDeletedTask_test_mode_eager_tfapiversion_1_numremoteworkers_0 [ RUN ] LocalWorkersTest.testAnonymousJobWithDifferentTargetWorkers_test_mode_graph_tfapiversion_2 INFO:tensorflow:Using local port 42591 I0723 21:52:18.372034 281473171159744 test_util.py:3813] Using local port 42591 2023-07-23 21:52:18.374034: I tensorflow/core/data/service/dispatcher_impl.cc:224] Attempting to restore dispatcher state from journal in /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/8a25e21c32908990f6c326230db25da5rfvvld5k/tmps9i0fs4i/tf_data_dispatcher_journal 2023-07-23 21:52:18.374108: I tensorflow/core/data/service/dispatcher_impl.cc:231] No journal found. Starting dispatcher from new state. 2023-07-23 21:52:18.374430: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data DispatchServer running at 0.0.0.0:42591 2023-07-23 21:52:18.374453: W tensorflow/core/data/service/dispatcher_impl.cc:1276] Error updating the optimal number of workers metric in tf.data service AutoScaler: UNAVAILABLE: Cannot update the optimal number of workers metric because there are no reported processing and target processing times for at least one iteration INFO:tensorflow:Using local port 32851 I0723 21:52:18.374776 281473171159744 test_util.py:3813] Using local port 32851 2023-07-23 21:52:18.395607: I tensorflow/core/data/service/worker_impl.cc:188] Worker registered with dispatcher running at localhost:42591 2023-07-23 21:52:18.395800: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:32851 INFO:tensorflow:Using local port 34961 I0723 21:52:18.396333 281473171159744 test_util.py:3813] Using local port 34961 2023-07-23 21:52:18.415222: I tensorflow/core/data/service/worker_impl.cc:188] Worker registered with dispatcher running at localhost:42591 2023-07-23 21:52:18.415467: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:34961 INFO:tensorflow:Using local port 41005 I0723 21:52:18.415985 281473171159744 test_util.py:3813] Using local port 41005 2023-07-23 21:52:18.418632: I tensorflow/core/data/service/worker_impl.cc:188] Worker registered with dispatcher running at localhost:42591 2023-07-23 21:52:18.418819: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:41005 INFO:tensorflow:Using local port 41323 I0723 21:52:18.428033 281473171159744 test_util.py:3813] Using local port 41323 2023-07-23 21:52:23.576994: I tensorflow/core/data/service/worker_impl.cc:188] Worker registered with dispatcher running at localhost:42591 2023-07-23 21:52:23.577268: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:41323 INFO:tensorflow:Using local port 42215 I0723 21:52:23.582058 281473171159744 test_util.py:3813] Using local port 42215 2023-07-23 21:52:23.647547: I tensorflow/core/data/service/worker_impl.cc:188] Worker registered with dispatcher running at localhost:42591 2023-07-23 21:52:23.647813: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:42215 INFO:tensorflow:Using local port 46851 I0723 21:52:23.648844 281473171159744 test_util.py:3813] Using local port 46851 2023-07-23 21:52:23.724133: I tensorflow/core/data/service/worker_impl.cc:188] Worker registered with dispatcher running at localhost:42591 2023-07-23 21:52:23.724521: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:46851 /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/data/ops/dataset_ops.py:458: UserWarning: To make it possible to preserve tf.data options across serialization boundaries, their implementation has moved to be part of the TensorFlow graph. As a consequence, the options value is in general no longer known at graph construction time. Invoking this method in graph mode retains the legacy behavior of the original implementation, but note that the returned value might not reflect the actual value of the options. warnings.warn("To make it possible to preserve tf.data options across " WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/contextlib.py:103: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `self.session()` or `self.cached_session()` instead. W0723 21:52:23.905848 281473171159744 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/contextlib.py:103: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `self.session()` or `self.cached_session()` instead. 2023-07-23 21:52:23.914129: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:382] MLIR V1 optimization pass is not enabled 2023-07-23 21:52:23.995685: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:24.084684: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:32851 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:24.428163: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2001 and consumer ID 3001 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:24.689414: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2001 and consumer ID 3001 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:24.908453: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:25.058762: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2002 and worker address localhost:42215 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:25.082613: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2001 and consumer ID 3001 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:25.100979: I tensorflow/core/data/service/server_lib.cc:94] Shut down WorkerServer server running at port 32851 2023-07-23 21:52:25.217699: I tensorflow/core/data/service/server_lib.cc:94] Shut down WorkerServer server running at port 34961 2023-07-23 21:52:25.231295: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2002 and consumer ID 3002 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:25.321895: I tensorflow/core/data/service/server_lib.cc:94] Shut down WorkerServer server running at port 41005 2023-07-23 21:52:26.392843: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2001 and consumer ID 3001 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:26.399959: I tensorflow/core/data/service/server_lib.cc:94] Shut down DispatchServer server running at port 42591 2023-07-23 21:52:26.417029: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3002. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.417173: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3001. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.417245: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3000. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.437408: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3002. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.437477: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3001. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.437545: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3000. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.457724: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3002. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.458471: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3000. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.458603: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3001. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.478694: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3002. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.486559: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3000. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.486777: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3001. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.506777: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3002. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.507053: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3000. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.507117: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3001. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.527284: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3002. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.527645: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3001. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.527672: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3000. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.547907: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3002. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.569245: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3001. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.569997: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3000. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.589590: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3001. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.590277: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3000. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.593541: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3002. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.609895: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3001. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.610567: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3000. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.613759: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3002. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.630202: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3001. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.630883: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3000. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.634000: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3002. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.650504: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3001. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.654296: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3002. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.658285: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3000. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.670780: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3001. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.674594: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3002. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.678606: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3000. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.691043: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3001. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.694882: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3002. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.698888: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3000. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.711305: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3001. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.715192: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3002. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.718499: W tensorflow/core/data/service/client/data_service_client.cc:90] Failed to release iteration client id: UNAVAILABLE: Failed to release iteration client with id 3002: failed to connect to all addresses 2023-07-23 21:52:26.719194: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3000. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.731596: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3001. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.738616: W tensorflow/core/data/service/client/data_service_client.cc:90] Failed to release iteration client id: UNAVAILABLE: Failed to release iteration client with id 3001: failed to connect to all addresses 2023-07-23 21:52:26.739489: W tensorflow/core/data/service/client/data_service_client.cc:439] Failed to heartbeat to dispatcher from iteration client id 3000. Dispatcher address: localhost:42591. Error: UNAVAILABLE: Failed to get tasks: failed to connect to all addresses 2023-07-23 21:52:26.748432: W tensorflow/core/data/service/client/data_service_client.cc:90] Failed to release iteration client id: UNAVAILABLE: Failed to release iteration client with id 3000: failed to connect to all addresses [ OK ] LocalWorkersTest.testAnonymousJobWithDifferentTargetWorkers_test_mode_graph_tfapiversion_2 [ RUN ] LocalWorkersTest.testEmptyDataset_test_mode_eager_tfapiversion_1_numlocalworkers_3_numremoteworkers_3 [ SKIPPED ] LocalWorkersTest.testEmptyDataset_test_mode_eager_tfapiversion_1_numlocalworkers_3_numremoteworkers_3 [ RUN ] LocalWorkersTest.testLocalWorkers_test_mode_eager_tfapiversion_2_numlocalworkers_3_numremoteworkers_3 INFO:tensorflow:Using local port 36487 I0723 21:52:26.760907 281473171159744 test_util.py:3813] Using local port 36487 2023-07-23 21:52:26.762559: I tensorflow/core/data/service/dispatcher_impl.cc:224] Attempting to restore dispatcher state from journal in /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/8a25e21c32908990f6c326230db25da5rfvvld5k/tmp4zjgvoxw/tf_data_dispatcher_journal 2023-07-23 21:52:26.762635: I tensorflow/core/data/service/dispatcher_impl.cc:231] No journal found. Starting dispatcher from new state. 2023-07-23 21:52:26.762911: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data DispatchServer running at 0.0.0.0:36487 INFO:tensorflow:Using local port 39459 I0723 21:52:26.763271 281473171159744 test_util.py:3813] Using local port 39459 2023-07-23 21:52:26.764275: W tensorflow/core/data/service/dispatcher_impl.cc:1276] Error updating the optimal number of workers metric in tf.data service AutoScaler: UNAVAILABLE: Cannot update the optimal number of workers metric because there are no reported processing and target processing times for at least one iteration 2023-07-23 21:52:26.765933: I tensorflow/core/data/service/worker_impl.cc:188] Worker registered with dispatcher running at localhost:36487 2023-07-23 21:52:26.766172: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:39459 INFO:tensorflow:Using local port 33817 I0723 21:52:26.766601 281473171159744 test_util.py:3813] Using local port 33817 2023-07-23 21:52:26.769345: I tensorflow/core/data/service/worker_impl.cc:188] Worker registered with dispatcher running at localhost:36487 2023-07-23 21:52:26.769545: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:33817 INFO:tensorflow:Using local port 46359 I0723 21:52:26.769947 281473171159744 test_util.py:3813] Using local port 46359 2023-07-23 21:52:26.777061: I tensorflow/core/data/service/worker_impl.cc:188] Worker registered with dispatcher running at localhost:36487 2023-07-23 21:52:26.777336: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:46359 INFO:tensorflow:Using local port 33607 I0723 21:52:26.777948 281473171159744 test_util.py:3813] Using local port 33607 2023-07-23 21:52:26.794551: I tensorflow/core/data/service/worker_impl.cc:188] Worker registered with dispatcher running at localhost:36487 2023-07-23 21:52:26.794918: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:33607 INFO:tensorflow:Using local port 33897 I0723 21:52:26.795930 281473171159744 test_util.py:3813] Using local port 33897 2023-07-23 21:52:26.813265: I tensorflow/core/data/service/worker_impl.cc:188] Worker registered with dispatcher running at localhost:36487 2023-07-23 21:52:26.813842: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:33897 INFO:tensorflow:Using local port 45213 I0723 21:52:26.814875 281473171159744 test_util.py:3813] Using local port 45213 2023-07-23 21:52:26.829942: I tensorflow/core/data/service/worker_impl.cc:188] Worker registered with dispatcher running at localhost:36487 2023-07-23 21:52:26.830325: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:45213 2023-07-23 21:52:27.735931: W tensorflow/core/data/service/dispatcher_impl.cc:746] Failed to remove consumer with ID 3000 for Iteration 2000 from tf.data service AutoScaler: NOT_FOUND: Consumer with ID 3000 not found 2023-07-23 21:52:27.847043: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:33897 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:27.857014: I tensorflow/core/data/service/server_lib.cc:94] Shut down WorkerServer server running at port 39459 2023-07-23 21:52:27.957633: I tensorflow/core/data/service/server_lib.cc:94] Shut down WorkerServer server running at port 33817 2023-07-23 21:52:28.066889: I tensorflow/core/data/service/server_lib.cc:94] Shut down WorkerServer server running at port 46359 2023-07-23 21:52:28.079989: I tensorflow/core/data/service/server_lib.cc:94] Shut down DispatchServer server running at port 36487 [ OK ] LocalWorkersTest.testLocalWorkers_test_mode_eager_tfapiversion_2_numlocalworkers_3_numremoteworkers_3 [ RUN ] LocalWorkersTest.testMultipleEpochs_test_mode_graph_tfapiversion_1_numlocalworkers_3_numremoteworkers_3 [ SKIPPED ] LocalWorkersTest.testMultipleEpochs_test_mode_graph_tfapiversion_1_numlocalworkers_3_numremoteworkers_3 [ RUN ] LocalWorkersTest.testNonLocalRead_test_mode_graph_tfapiversion_2_numlocalworkers_3_numremoteworkers_3 INFO:tensorflow:Using local port 42977 I0723 21:52:28.811787 281473171159744 test_util.py:3813] Using local port 42977 2023-07-23 21:52:28.813284: I tensorflow/core/data/service/dispatcher_impl.cc:224] Attempting to restore dispatcher state from journal in /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/8a25e21c32908990f6c326230db25da5rfvvld5k/tmpgo69h6tl/tf_data_dispatcher_journal 2023-07-23 21:52:28.813366: I tensorflow/core/data/service/dispatcher_impl.cc:231] No journal found. Starting dispatcher from new state. 2023-07-23 21:52:28.814049: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data DispatchServer running at 0.0.0.0:42977 2023-07-23 21:52:28.814068: W tensorflow/core/data/service/dispatcher_impl.cc:1276] Error updating the optimal number of workers metric in tf.data service AutoScaler: UNAVAILABLE: Cannot update the optimal number of workers metric because there are no reported processing and target processing times for at least one iteration INFO:tensorflow:Using local port 39993 I0723 21:52:28.814402 281473171159744 test_util.py:3813] Using local port 39993 2023-07-23 21:52:28.816712: I tensorflow/core/data/service/worker_impl.cc:188] Worker registered with dispatcher running at localhost:42977 2023-07-23 21:52:28.816917: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:39993 INFO:tensorflow:Using local port 33795 I0723 21:52:28.817295 281473171159744 test_util.py:3813] Using local port 33795 2023-07-23 21:52:28.819214: I tensorflow/core/data/service/worker_impl.cc:188] Worker registered with dispatcher running at localhost:42977 2023-07-23 21:52:28.819428: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:33795 INFO:tensorflow:Using local port 32859 I0723 21:52:28.819800 281473171159744 test_util.py:3813] Using local port 32859 2023-07-23 21:52:28.823249: I tensorflow/core/data/service/worker_impl.cc:188] Worker registered with dispatcher running at localhost:42977 2023-07-23 21:52:28.823486: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:32859 INFO:tensorflow:Using local port 37281 I0723 21:52:28.824120 281473171159744 test_util.py:3813] Using local port 37281 2023-07-23 21:52:28.839658: I tensorflow/core/data/service/worker_impl.cc:188] Worker registered with dispatcher running at localhost:42977 2023-07-23 21:52:28.840031: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:37281 INFO:tensorflow:Using local port 40925 I0723 21:52:28.841006 281473171159744 test_util.py:3813] Using local port 40925 2023-07-23 21:52:28.856808: I tensorflow/core/data/service/worker_impl.cc:188] Worker registered with dispatcher running at localhost:42977 2023-07-23 21:52:28.857189: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:40925 INFO:tensorflow:Using local port 38523 I0723 21:52:28.858276 281473171159744 test_util.py:3813] Using local port 38523 2023-07-23 21:52:28.880636: I tensorflow/core/data/service/worker_impl.cc:188] Worker registered with dispatcher running at localhost:42977 2023-07-23 21:52:28.881039: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:38523 2023-07-23 21:52:29.362684: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:30.231005: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:30.231051: W tensorflow/core/data/service/dispatcher_impl.cc:367] Could not find task with id 4008 in tf.data service dispatcher state: NOT_FOUND: Task 4008 not found 2023-07-23 21:52:30.331593: W tensorflow/core/data/service/dispatcher_impl.cc:367] Could not find task with id 4012 in tf.data service dispatcher state: NOT_FOUND: Task 4012 not found 2023-07-23 21:52:30.353763: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:30.535519: W tensorflow/core/data/service/dispatcher_impl.cc:367] Could not find task with id 4016 in tf.data service dispatcher state: NOT_FOUND: Task 4016 not found 2023-07-23 21:52:30.638264: W tensorflow/core/data/service/dispatcher_impl.cc:367] Could not find task with id 4020 in tf.data service dispatcher state: NOT_FOUND: Task 4020 not found 2023-07-23 21:52:30.779385: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:31.606356: W tensorflow/core/data/service/dispatcher_impl.cc:367] Could not find task with id 4024 in tf.data service dispatcher state: NOT_FOUND: Task 4024 not found 2023-07-23 21:52:32.270778: W tensorflow/core/data/service/dispatcher_impl.cc:367] Could not find task with id 4028 in tf.data service dispatcher state: NOT_FOUND: Task 4028 not found 2023-07-23 21:52:32.396248: W tensorflow/core/data/service/dispatcher_impl.cc:367] Could not find task with id 4032 in tf.data service dispatcher state: NOT_FOUND: Task 4032 not found 2023-07-23 21:52:32.998836: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:33.125100: W tensorflow/core/data/service/dispatcher_impl.cc:367] Could not find task with id 4036 in tf.data service dispatcher state: NOT_FOUND: Task 4036 not found 2023-07-23 21:52:33.498843: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:34.078649: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:34.091648: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:34.543871: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:35.086628: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:35.715386: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:36.205934: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:36.318763: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:36.648228: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:37.063072: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:37.479271: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:37.892496: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:38.430163: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:39.158562: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:39.580706: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:40.089239: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:41.427373: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:41.840799: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:42.167784: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:42.666606: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:43.199887: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:43.647488: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:44.258658: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:44.355540: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:44.834983: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:45.358826: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:45.788531: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:46.916409: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:47.083582: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:47.404582: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:47.873387: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:49.580901: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:50.187045: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:50.449486: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:50.741576: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:51.451594: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:51.888966: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:53.080929: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:53.367850: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:53.588242: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:54.060655: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:54.527965: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:54.977052: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:55.456381: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:55.479966: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:55.868609: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:56.373236: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:56.790919: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:57.227898: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:57.697972: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:57.702621: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:58.247394: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:58.866637: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:59.319402: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:59.744907: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:52:59.822327: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:00.158084: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:00.658683: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:01.340196: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:02.326544: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:02.387957: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:02.768560: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:03.187599: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:03.603600: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:04.152827: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:04.430508: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:04.584894: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:05.106778: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:05.546760: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:05.963436: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:06.378361: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:06.481293: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:06.829635: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:07.647528: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:08.098477: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:08.536377: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:08.856673: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:09.000265: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:09.567043: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:10.207381: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:10.756519: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:11.066945: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:11.396377: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:12.097441: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:12.786533: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:13.257839: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:13.346942: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:13.932327: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:14.478352: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:15.076545: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:15.444062: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:15.648886: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:16.284713: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:16.807962: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:17.397417: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:17.576798: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:17.946673: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:18.428232: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:18.976318: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:19.586491: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:19.777149: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:20.266339: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:20.916988: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:21.576308: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:22.084927: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:22.146484: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:22.816302: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:23.557918: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:24.206301: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:24.347784: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:24.767465: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:25.446727: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:26.048420: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:26.536543: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:26.559675: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:27.098461: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:27.677192: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:28.227007: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:28.758055: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:28.877559: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:29.466673: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:30.054566: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:30.626689: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:30.907567: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:31.246557: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:31.798739: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:32.285286: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:32.946699: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:33.108342: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:33.566659: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:34.152030: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:34.729438: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:35.261470: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:35.269109: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:35.936668: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:36.556486: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:37.356822: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:37.646237: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:38.127447: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:38.766625: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:39.336642: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:39.877398: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:39.966584: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:40.596628: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:41.188930: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:41.806572: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:42.037299: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:42.456548: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:43.099159: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:43.758274: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:44.217646: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:44.412942: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:45.027447: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:45.766818: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:46.427025: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:46.505129: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:47.067129: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:47.696635: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:48.318129: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:48.699021: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:48.835301: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:49.516566: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:50.108447: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:50.684658: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:50.888362: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:51.277289: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:51.936620: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:52.546596: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:53.147432: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:53.187063: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:53.789732: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:54.376805: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:55.008386: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:55.357800: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:55.686652: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:56.278289: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:56.835668: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:57.467707: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:57.567452: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:58.047495: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:58.466907: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:59.066577: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:59.687647: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:53:59.777016: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:00.296607: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:00.956617: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:01.552866: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:01.978210: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:02.126358: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:02.831479: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:03.408627: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:04.066347: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:04.184061: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:04.598290: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:05.245800: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:05.844961: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:06.358907: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:06.388380: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:06.927776: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:07.547697: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:08.156317: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:08.474389: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:08.786619: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:09.387453: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:09.856699: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:10.400110: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:10.640910: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:10.997205: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:11.618020: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:12.244161: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:12.778724: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:12.856795: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:13.466618: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:14.077378: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:14.634178: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:14.987532: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:15.116905: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:15.717126: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:16.256634: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:16.856643: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:17.187153: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:17.456579: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:18.027959: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:18.616606: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:19.230513: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:19.356492: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:19.801543: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:20.386924: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:20.946395: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:21.466404: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:21.467049: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:22.056618: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:22.606543: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:23.193762: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:23.637593: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:23.787093: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:24.378358: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:24.950612: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:25.509198: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:25.737432: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:26.137053: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:26.612211: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:27.163825: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:27.776792: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:27.947605: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:28.426317: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:29.078045: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:29.648436: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:30.197312: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:30.226364: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:30.794523: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:31.307437: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:31.916536: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:32.407147: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:32.537149: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:33.176851: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:33.786954: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:34.336733: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:34.706364: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:34.927175: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:35.617354: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:36.196796: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:36.782746: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:36.898379: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:37.456366: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:38.007497: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:38.583132: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:39.086491: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:39.107320: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:39.746399: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:40.290886: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:40.916331: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:41.366655: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:41.476829: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:42.086302: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:42.686293: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:43.228130: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:43.586256: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:43.726302: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:44.286809: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:44.936326: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:45.506355: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:45.876392: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:46.126314: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:46.676289: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:47.226288: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:47.786297: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:48.086390: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:48.436256: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:49.136670: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:49.707523: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:50.277894: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:50.387156: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:50.896370: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:51.405778: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:51.957987: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:52.586363: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:52.647223: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:53.246333: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:54.086340: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:54.666478: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:54.936917: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:55.199837: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:55.609459: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:56.016769: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:56.423622: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:56.830486: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:56.963985: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:57.238315: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:57.645296: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:58.052250: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:58.458865: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:58.865859: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:58.970827: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:59.319023: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:54:59.726101: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:00.133959: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:00.541248: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:00.948876: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:00.982802: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:01.357103: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:01.764182: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:02.172537: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:02.580863: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:03.026957: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:03.029797: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:03.470737: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:03.877852: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:04.285511: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:04.750278: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:05.096571: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:05.182993: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:05.589967: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:06.056833: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:06.464682: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:06.874774: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:07.121378: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:07.285124: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:07.777299: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:08.184870: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:08.593310: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:09.079646: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:09.196486: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:09.781669: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:10.317977: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:10.877188: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:11.390436: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:11.456594: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:12.016496: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:12.686426: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:13.106657: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:13.511983: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:13.690279: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:14.136668: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:14.668062: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:15.166921: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:15.637527: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:15.672477: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:16.141164: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:16.554054: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:16.966249: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:17.404809: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:17.699875: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:17.827821: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:18.248891: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:18.688167: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:19.119994: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:19.532795: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:20.254274: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:20.568952: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:21.007380: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:21.782984: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:22.511905: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:23.166273: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:23.378384: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:23.786104: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:24.393554: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:25.266895: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:25.660552: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:26.016277: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:26.440669: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:27.256291: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:28.010785: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:28.466942: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:29.058492: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:29.526977: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:30.088191: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:30.586540: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:31.027821: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:31.117456: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:31.607815: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:32.076405: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:32.722424: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:33.197165: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:33.359900: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:33.873455: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:34.324398: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:34.813314: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:35.346658: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:35.376758: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:35.921857: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:36.396001: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:36.852956: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:37.299063: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:37.543195: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:37.754100: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:38.248164: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:38.784003: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:39.317403: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:39.702014: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:39.766579: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:40.307557: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:40.717421: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:41.125097: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:41.534094: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:41.773824: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:41.941817: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:42.350145: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:42.832187: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:43.244696: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:43.848726: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:44.303375: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:44.723595: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:45.965063: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:46.793063: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:47.201790: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:47.388979: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:47.740108: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:48.385102: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:48.807573: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:49.849370: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:50.128928: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:50.660245: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:51.198217: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:51.613256: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:52.324156: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:52.739351: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:52.815682: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:53.192201: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:53.605548: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:54.019842: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:54.434116: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:54.830269: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:54.848811: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:55.316308: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:55.793411: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:56.206590: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:56.621507: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:56.865475: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:57.037760: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:57.454003: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:57.896638: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:58.322335: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:58.738134: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:58.877830: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:59.152499: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:59.568312: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:55:59.983260: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:00.398327: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:00.841547: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:00.898233: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:01.285224: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:01.699482: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:02.131355: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:02.546355: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:02.929586: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:03.017424: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:03.430746: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:03.845116: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:04.259193: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:04.672316: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:04.948700: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:05.087595: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:05.502675: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:05.918784: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:06.334506: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:06.763606: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:06.961881: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:07.210885: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:07.626500: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:08.039974: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:08.455713: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:08.873025: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:08.973432: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:09.823672: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:10.310355: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:10.751035: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:11.236268: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:11.625869: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:11.650909: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:12.092821: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:12.507127: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:13.008314: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:13.441794: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:13.737518: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:13.969520: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:14.388407: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:14.866728: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:15.576757: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:15.957683: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:16.217271: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:16.836796: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:17.387901: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:17.850281: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:18.223053: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:18.515234: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:19.016400: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:19.514401: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:20.029403: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:20.360456: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:20.496723: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:21.056416: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:21.777982: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:22.268709: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:22.617442: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:22.808904: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:23.357515: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:23.965409: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:24.519218: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:24.786991: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:25.067861: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:25.558026: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:26.085113: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:26.580181: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:26.939755: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:27.085197: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:27.656999: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:28.147137: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:28.696416: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:29.109390: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:29.250024: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:29.758024: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:30.295361: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:30.806719: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:31.216382: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:31.256969: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:31.716747: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:32.229780: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:32.713890: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:33.215183: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:33.319237: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:33.748452: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:34.267429: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:34.814597: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:35.376770: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:35.468420: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:35.899359: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:36.410797: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:36.908183: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:37.469597: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:37.633513: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:38.027082: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:38.531084: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:39.011885: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:39.435109: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:40.478814: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:42.627319: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:43.044124: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:43.466803: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:44.131484: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:45.280473: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:45.315057: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:45.724711: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:46.602101: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:47.038728: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:47.701381: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:48.166714: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:48.545792: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:49.084977: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:49.636255: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:50.070984: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:50.287980: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:50.484721: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:50.957581: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:51.373961: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:51.789959: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:52.206053: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:52.451005: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:52.766254: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:53.265918: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:53.680613: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:54.096069: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:54.505878: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:54.509771: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:54.946998: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:55.359739: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:55.845035: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:56.296641: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:56.538809: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:56.771892: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:57.185902: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:57.602352: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:58.019120: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:58.434065: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:58.563020: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:58.936635: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:59.383226: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:56:59.796915: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:00.210124: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:00.654125: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:00.656665: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:01.068855: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:01.482196: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:01.911922: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:02.350394: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:02.714106: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:02.797639: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:03.289834: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:03.767365: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:04.226674: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:04.698403: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:04.806511: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:05.275898: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:05.838182: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:06.338632: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:06.797054: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:06.971690: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:07.251453: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:07.689038: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:08.166453: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:08.689675: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:09.060206: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:09.175450: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:09.612322: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:10.046712: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:10.516990: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:10.973499: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:11.133747: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:11.422271: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:11.846473: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:12.336517: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:12.874859: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:13.205441: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:13.309036: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:13.730722: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:14.146892: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:14.563576: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:23.290011: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:26.806863: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:27.113115: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:28.319933: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:30.142720: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:30.649166: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:30.849458: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:32.574070: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:33.055050: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:33.600925: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:34.149075: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:34.419664: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:34.564879: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:34.978990: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:35.392860: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:35.806191: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:36.219372: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:36.433509: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:36.634225: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:37.047035: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:37.502568: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:37.956361: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:38.453645: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:38.556757: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:38.996735: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:39.466652: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:39.891064: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:40.363645: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:40.631201: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:40.831380: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:41.244514: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:41.655879: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:42.075268: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:43.606342: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:43.687553: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:44.058965: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:44.469867: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:45.047345: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:45.492380: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:47.176686: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:47.288971: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:47.739377: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:48.204066: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:48.628086: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:49.459841: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:50.757372: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:51.172441: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:51.647160: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:52.059048: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:52.480917: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:53.271053: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:53.383354: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:53.798716: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:54.293815: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:54.705937: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:55.267303: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:55.354124: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:55.684049: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:56.183114: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:57.652206: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:58.069098: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:58.345010: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:57:59.572156: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:00.037572: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:00.595310: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:01.011323: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:01.426105: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:01.522412: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:01.842276: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:02.301404: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:02.866418: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:03.283410: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:03.643117: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:03.700031: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:04.196860: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:04.963714: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:05.411259: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:05.827683: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:05.973155: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:06.242933: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:06.658926: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:07.107473: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:07.552101: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:08.015883: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:08.033045: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:08.432303: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:08.845873: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:09.263018: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:09.706465: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:10.050572: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:10.180489: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:10.710686: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:11.256743: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:11.811803: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:12.157514: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:12.386447: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:12.847037: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:13.298825: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:13.753375: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:14.207477: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:14.289419: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:14.664274: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:15.190072: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:15.629768: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:16.097895: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:16.401966: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:16.563905: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:17.051873: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:17.469627: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:17.930747: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:18.344158: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:18.434406: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:18.844752: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:19.260866: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:20.351156: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:21.369897: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:21.554369: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:21.839306: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:22.744585: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:23.162687: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:23.581748: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:25.207750: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:25.256570: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:25.677576: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:26.136620: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:26.657157: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:27.069167: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:27.318996: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:27.507529: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:27.923280: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:28.336552: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:28.833598: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:29.249075: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:29.349543: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:29.662288: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:30.127093: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:30.562517: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:30.978800: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:31.386413: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:31.428782: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:31.842663: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:32.255087: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:32.670345: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:33.085436: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:33.400538: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:33.522343: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:33.975663: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:34.470138: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:34.885106: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:35.338504: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:35.443716: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:35.795011: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:36.210730: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:36.625962: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:37.039533: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:37.451237: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:37.576966: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:37.990952: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:38.405454: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:38.947598: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:40.011547: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:40.180050: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:40.427256: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:41.430275: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:41.886505: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:42.365266: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:42.766538: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:42.839469: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:43.254130: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:43.672788: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:44.089978: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:44.563160: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:44.825199: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:45.066362: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:45.583511: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:45.997979: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:46.413854: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:46.828540: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:46.880892: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:47.299661: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:47.713227: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:48.475348: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:48.948938: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:49.171252: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:49.362290: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:49.976969: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:50.398694: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:50.815648: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:51.231009: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:51.374589: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:51.708883: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:52.156260: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:52.601946: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:53.011599: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:53.917246: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:54.031950: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:54.447653: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:54.894758: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:55.349211: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:55.805315: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:55.950470: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:56.223437: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:56.639240: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:57.099152: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:57.513573: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:58.067273: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:58.077906: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:58:58.523830: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:00.146627: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:00.588265: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:01.089120: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:01.279189: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:01.504726: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:01.948822: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:02.364901: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:02.778826: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:03.193563: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:03.291552: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:03.650726: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:04.065322: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:04.560090: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:04.974200: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:05.349524: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:05.389027: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:05.801055: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:06.214928: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:06.673117: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:07.086303: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:07.372699: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:07.892426: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:08.363457: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:08.813998: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:09.230241: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:09.666435: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:09.788736: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:10.120697: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:10.535147: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:10.951370: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:11.367917: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:11.787258: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:11.821187: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:12.238014: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:12.656253: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:13.160444: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:13.699224: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:13.967227: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:14.212915: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:14.725922: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:15.140016: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:15.556311: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:16.011723: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:16.101846: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:16.429743: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:16.868685: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:17.324143: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:17.758747: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:18.125355: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:18.177894: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:18.636694: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:19.066383: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:19.485071: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:19.931066: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:20.216422: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:20.487120: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:21.068807: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:21.547640: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:22.004782: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:22.363226: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:22.527510: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:22.991108: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:23.423475: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:23.853024: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:24.305192: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:24.411798: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:24.795945: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:25.249677: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:25.723569: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:26.160341: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:26.490611: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:26.583502: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:27.117551: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:27.696795: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:28.198240: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:28.611353: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:28.909427: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:29.342954: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:30.235834: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:31.616322: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:32.076319: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:32.076670: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:32.556833: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:32.980697: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:33.393677: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:33.827123: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:34.146406: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:34.420283: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:34.838992: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:35.306145: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:35.836471: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:36.187584: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:36.334801: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:36.926770: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:37.372687: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:37.807795: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:38.238487: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:38.297069: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:38.658392: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:39.111369: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:39.680747: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:40.129128: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:40.331595: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:40.551939: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:41.010158: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:41.436494: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:41.908692: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:42.436456: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:42.440476: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:42.891612: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:43.315232: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:43.808934: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:44.262197: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:44.506440: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:44.806361: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:45.266285: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:45.870580: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:46.305949: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:46.622315: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:46.761541: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 21:59:47.170005: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:03.486638: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:04.048113: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:04.526765: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:04.846629: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:05.428422: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:05.879950: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:06.353697: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:06.639638: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:06.811004: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:07.315230: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:07.800094: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:08.244061: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:08.701670: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:08.740344: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:09.226787: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:09.658846: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:10.115861: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:10.662887: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:10.838338: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:11.109249: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:11.636394: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:12.266864: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:12.796677: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:13.023454: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:13.376648: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:13.906653: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:14.392537: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:14.886533: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:15.106474: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:15.346679: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:15.786413: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:16.267505: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:16.786376: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:17.239578: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:17.277698: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:17.774237: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:18.243905: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:18.704778: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:19.162774: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:19.325719: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:19.611017: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:20.086968: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:20.567719: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:21.002091: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:21.448800: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:21.500177: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:21.936061: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:22.426673: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:22.914853: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:23.364608: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:23.534223: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:23.904444: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:24.492089: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:24.931422: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:25.391431: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:25.637031: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:25.927612: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:26.470693: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:26.931832: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:27.386787: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:27.815485: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:27.828054: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:28.407139: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:28.891422: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:29.356622: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:29.837943: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:29.969331: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:30.269245: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:30.707076: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:31.131461: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:31.618382: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:32.019551: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:32.103912: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:32.726331: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:33.228559: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:33.807338: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:34.236786: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:34.393807: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:34.967862: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:35.460800: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:36.038183: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:36.367553: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:36.627062: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:37.147122: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:37.682565: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:38.102437: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:38.417592: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:38.606862: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:39.254406: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:39.675578: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:40.158819: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:40.586640: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:40.708813: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:41.266838: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:41.758154: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:42.211428: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:42.630658: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:42.630971: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:43.100923: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:43.656706: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:44.330899: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:44.786583: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:44.936358: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:45.290808: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:45.722947: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:46.205665: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:46.666473: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:47.088164: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:47.122697: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:47.560285: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:48.463406: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:49.009110: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:49.546597: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:49.603380: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:49.966652: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:50.481314: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:50.915850: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:51.365095: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:51.677224: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:51.782878: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:52.198904: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:52.619575: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:53.186926: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:53.736701: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:53.776774: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:54.211563: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:54.856418: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:55.398357: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:55.967026: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:55.974546: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:56.556632: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:57.342918: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:57.897497: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:58.306624: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:58.472219: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:59.090950: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:00:59.607889: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:00.150398: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:00.637320: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:00.766499: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:01.370238: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:01.910347: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:02.366336: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:02.801439: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:02.923208: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:03.471521: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:04.074804: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:04.627691: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:04.957190: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:05.211791: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:05.790993: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:06.379286: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:07.006428: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:07.118695: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:07.776430: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:08.341490: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:08.832804: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:09.324661: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:09.397030: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:09.929114: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:10.356024: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:10.787290: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:11.234110: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:11.424621: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:11.785898: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:12.306456: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:12.767993: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:13.221203: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:13.815116: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:14.045699: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:14.459732: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:15.447355: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:15.959926: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:16.330565: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:16.388935: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:16.838618: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:17.276084: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:17.704795: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:18.161668: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:18.358238: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:18.687574: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:19.146834: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:19.632895: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:20.078665: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:20.474429: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:20.505785: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:20.935600: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:21.368852: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:21.786309: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:22.244555: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:22.557636: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:22.677017: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:23.152056: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:23.987043: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:24.456857: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:24.912519: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:24.947148: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:25.382913: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:25.867506: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:26.315932: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:26.808754: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:26.993975: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:27.233492: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:27.688890: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:28.109326: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:28.584057: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:29.052026: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:29.063038: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:29.526539: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:29.946143: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:30.362920: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:30.794509: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:31.129260: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:31.297616: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:31.808136: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:32.271190: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:32.717221: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:33.193325: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:33.193746: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:33.621888: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:34.108842: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:34.565767: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:35.016155: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:35.246429: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:35.464853: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:35.908186: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:36.352869: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:36.795498: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:37.214485: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:37.338468: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:37.692734: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:38.166688: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:38.656420: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:39.159089: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:39.446918: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:39.632397: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:40.164218: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:40.630235: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:41.086607: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:41.560309: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:41.626386: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:42.119636: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:42.617147: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:43.142966: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:43.649457: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:43.756349: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:44.217066: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:44.753907: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:45.226223: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:45.650182: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:45.865846: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:46.177362: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:46.633515: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:47.094794: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:47.601883: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:47.916856: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:48.107464: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:48.541967: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:48.957736: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:49.391618: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:49.810852: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:50.040513: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:50.289293: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:50.730775: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:51.146486: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:51.567161: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:51.977600: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:52.807991: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:53.197384: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:53.633545: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:54.395629: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:55.189353: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:55.488914: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:55.716574: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:56.176303: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:56.640070: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:57.070003: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:57.491788: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:57.588069: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:57.931912: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:58.404406: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:58.874754: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:59.293213: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:59.666619: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:01:59.728348: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:00.142431: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:00.556643: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:00.999156: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:01.436666: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:01.738220: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:01.854180: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:02.282975: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:03.018921: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:03.497639: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:03.927939: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:03.995285: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:04.438722: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:04.867442: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:05.296044: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:05.717496: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:06.100021: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:06.154192: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:06.590439: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:07.136781: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:07.656648: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:08.092584: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:08.208354: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:08.542847: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:08.997749: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:09.542516: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:10.072631: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:10.268450: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:10.498289: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:10.966704: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:11.441836: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:11.958890: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:12.336445: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:12.430154: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:12.906756: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:13.435822: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:13.943003: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:14.397122: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:14.476422: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:14.829851: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:15.326034: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:15.763751: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:16.226676: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:16.547324: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:16.789978: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:17.230803: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:17.666683: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:18.104293: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:18.543554: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:18.616575: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:18.991654: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:19.425350: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:19.877522: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:20.293421: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:20.666856: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:20.809128: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:21.263911: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:21.740422: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:22.173329: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:22.593855: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:22.730119: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:23.010194: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:23.515114: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:23.998897: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:24.471814: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:24.816554: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:24.983734: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:26.106561: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:26.633049: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:27.126513: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:27.494584: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:27.606404: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:28.077062: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:28.631687: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:28.814234: W tensorflow/core/data/service/dispatcher_impl.cc:1276] Error updating the optimal number of workers metric in tf.data service AutoScaler: UNAVAILABLE: Cannot update the optimal number of workers metric because there are no reported processing and target processing times for at least one iteration 2023-07-23 22:02:29.159096: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:29.584046: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:29.611870: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:30.015571: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:30.464567: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:30.972806: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:31.457783: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:31.706758: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:31.902991: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:32.341876: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:32.857501: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:33.327209: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:33.798020: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:33.892440: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:34.366366: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:34.858117: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:35.297551: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:35.766363: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:35.887712: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:36.230925: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:36.867482: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:37.380721: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:37.793889: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:38.012445: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:38.207249: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:38.626154: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:39.068462: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:39.500633: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:40.530774: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:40.549060: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:41.046617: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:41.584244: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:42.087975: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:42.535787: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:42.656627: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:43.057017: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:43.616379: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:44.186393: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:44.766355: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:44.936962: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:45.294644: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:45.779142: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:46.379917: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:46.856351: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:47.046405: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:47.339001: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:47.937826: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:48.421830: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:48.918987: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:49.196488: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:49.348155: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:49.777263: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:50.459422: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:51.256491: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:51.457945: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:51.986539: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:52.636594: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:53.256760: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:53.630371: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:53.806598: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:54.381645: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:54.928499: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:55.573409: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:55.797863: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:56.096370: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:56.579581: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:57.036523: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:02:57.502113: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:05.839750: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:05.936467: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:06.427884: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:06.868707: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:07.332682: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:23.399376: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:29.933104: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:30.416626: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:30.979546: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:31.520885: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:31.996447: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:32.166722: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:32.417556: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:32.835830: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:33.282226: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:33.716416: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:34.189589: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:34.247561: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:34.654806: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:35.071652: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:35.494914: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:35.911356: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:36.281344: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:36.375209: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:36.795548: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:37.215464: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:37.672200: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:38.090009: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:38.346700: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:38.540485: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:39.010658: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:39.533783: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:40.046006: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:40.515111: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:40.519841: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:40.964249: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:41.387534: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:41.846614: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:42.346538: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:42.582444: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:42.880064: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:43.363556: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:43.821212: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:44.240102: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:44.633043: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:44.659160: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:45.077071: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:45.512045: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:45.929701: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:46.430547: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:46.728163: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:46.886209: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:47.315902: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:47.729279: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:48.156063: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:48.617300: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:48.800287: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:49.153045: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:49.677903: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:50.174347: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:50.609777: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:50.908183: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:51.029193: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:51.488466: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:52.037722: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:52.506926: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:53.047278: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:53.078094: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:53.608149: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:54.096957: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:54.648357: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:55.199402: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:55.266571: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:55.754652: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:56.367123: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:57.106536: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:57.369899: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:57.728411: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:58.536533: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:59.172576: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:59.656593: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:03:59.827212: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:00.394175: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:01.279977: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:01.833645: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:02.207268: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:02.645705: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:03.229554: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:03.786768: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:04.577068: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:04.877637: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:05.308654: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:05.794174: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:06.386284: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:06.986828: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:07.246283: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:08.024510: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:08.926591: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:09.510820: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:09.515949: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:10.077004: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:10.670842: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:11.376287: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:11.798495: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:12.056359: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:12.680889: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:13.387811: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:13.889473: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:13.997631: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:14.442718: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:14.989431: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:15.566891: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:16.127124: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:16.127648: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:16.806507: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:17.976517: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:18.824188: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:18.879244: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:19.398275: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:19.891464: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:20.390580: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:20.933702: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:21.006648: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:21.506601: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:22.061113: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:22.595234: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:23.357134: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:23.477136: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:24.246494: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:25.066351: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:25.889014: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:25.899207: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:26.447271: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:27.182056: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:27.803020: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:28.388692: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:28.393101: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:29.026394: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:30.011307: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:30.858824: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:30.996937: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:31.836261: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:32.537176: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:33.206775: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:33.416441: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:33.688564: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:34.379188: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:34.955370: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:35.434186: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:35.774039: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:35.956583: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:36.524504: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:37.047129: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:37.661925: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:37.947468: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:38.346307: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:38.986240: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:39.612824: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:40.286166: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:40.291790: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:40.835875: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:41.414610: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:42.056453: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:42.594734: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:42.615664: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:43.206543: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:43.837270: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:44.477168: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:44.926763: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:45.216496: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:45.869206: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:46.406462: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:46.927302: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:47.154238: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:47.616700: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:48.536856: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:49.001878: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:49.543059: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:49.572195: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:50.086121: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:50.726949: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:51.497118: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:51.719083: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:52.079645: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:52.650147: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:53.157032: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:53.753065: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:53.947039: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:54.967273: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:55.676508: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:56.291774: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:56.596703: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:56.758264: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:57.246409: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:57.757436: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:58.396390: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:58.757231: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:58.926628: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:04:59.506474: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:00.100058: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:00.665193: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:00.946962: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:01.236564: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:02.030465: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:02.856619: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:03.356648: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:03.556632: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:04.127991: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:04.716592: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:05.216727: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:05.616909: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:05.806524: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:06.366596: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:06.996572: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:07.556570: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:07.807417: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:08.114310: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:08.806663: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:09.336647: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:09.917903: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:10.046503: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:10.486920: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:11.178204: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:11.849862: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:12.261312: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:12.387258: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:13.004751: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:13.566651: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:14.078147: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:14.426848: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:14.594941: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:15.142354: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:15.735250: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:16.376567: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:16.705670: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:16.899922: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:17.456356: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:18.034919: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:18.606387: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:18.840978: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:19.182105: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:19.776623: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:20.546571: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:21.137255: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:21.146424: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:21.826613: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:22.736510: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:23.411342: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:23.626966: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:24.006371: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:24.564817: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:25.177779: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:25.766432: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:25.842321: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:26.335616: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:26.956511: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:27.557749: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:28.037861: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:28.116674: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:28.646352: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:29.256423: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:29.867037: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:30.247714: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:30.501959: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:31.066438: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:31.696286: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:32.310751: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:32.446401: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:32.867171: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:33.440554: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:34.036280: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:34.747746: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:34.846533: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:35.515285: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:36.177619: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:37.647092: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:37.718783: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:38.306625: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:38.906731: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:39.481831: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:39.888300: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:40.151687: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:40.806356: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:41.457043: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:42.070754: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:54.675533: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:55.373403: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:56.007249: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:56.584779: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:56.855169: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:05:57.116406: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:09.753629: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:10.776062: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:11.256419: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:11.671509: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:11.767270: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:12.256319: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:12.718619: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:13.216623: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:13.798167: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:13.909527: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:14.521657: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:15.077681: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:15.613494: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:15.956928: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:16.168347: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:16.770838: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:17.503250: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:18.146571: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:18.176795: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:18.808214: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:19.278657: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:19.754650: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:20.226017: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:20.376758: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:20.746574: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:21.218808: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:21.706464: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:22.224264: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:22.494482: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:22.726044: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:23.206353: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:23.653412: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:24.109382: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:24.551487: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:24.558452: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:24.990959: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:25.437621: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:25.902402: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:26.350804: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:26.617457: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:26.785096: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:27.210977: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:27.648036: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:28.099143: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:28.524013: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:28.669119: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:29.006458: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:29.433581: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:29.855708: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:31.615084: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:31.995503: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:32.045568: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:32.477481: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:32.904490: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:33.340753: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:33.811232: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:34.045826: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:34.234811: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:34.676684: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:35.157817: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:35.682347: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:36.127061: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:36.180612: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:36.656653: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:37.215138: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:37.759667: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:38.217128: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:38.306399: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:38.711625: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:39.199186: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:39.695000: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:40.171858: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:40.382352: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:40.618915: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:41.075851: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:41.504738: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:42.261892: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:42.867552: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:42.880869: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:43.371296: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:43.784843: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:44.197509: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:44.772794: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:45.026752: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:45.229419: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:45.646518: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:46.082029: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:46.531467: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:47.007090: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:47.066901: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:47.460151: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:47.886870: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:48.336962: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:48.769412: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:49.103632: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:49.219578: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:49.636445: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:50.052924: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:50.468801: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:50.896646: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:51.159267: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:51.381682: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:51.803040: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:52.219866: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:52.652384: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:53.069288: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:53.197235: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:53.518999: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:54.056650: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:54.516478: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:54.976814: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:55.278875: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:55.397064: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:55.851058: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:56.318208: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:56.772207: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:57.188166: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:57.366998: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:57.609774: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:58.054237: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:58.546932: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:59.020434: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:59.394250: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:59.441693: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:06:59.890694: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:07:00.309126: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:07:00.725350: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:07:01.192298: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:07:01.498265: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:07:01.751294: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:07:02.202435: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:07:02.619089: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:07:03.069760: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:07:03.509712: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:07:03.575329: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:07:03.924806: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:07:04.338360: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:07:04.760334: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:07:05.174622: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:07:05.590313: W tensorflow/core/data/service/dispatcher_impl.cc:376] Failed to report processing time for Iteration 2000 and worker address localhost:37281 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:07:05.592290: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 2023-07-23 22:07:06.008411: W tensorflow/core/data/service/dispatcher_impl.cc:1085] Failed to report target processing time for Iteration 2000 and consumer ID 3000 to tf.data service AutoScaler: INVALID_ARGUMENT: Cannot update target_processing_time with a ZeroDuration or negative value: 0 -- Test timed out at 2023-07-23 22:07:06 UTC -- Current thread 0x0000ffff94617ac0 (most recent call first): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1478 in _call_tf_sessionrun File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1385 in _run_fn File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1402 in _do_call File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1395 in _do_run File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1215 in _run File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 972 in run File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 2066 in run File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 2706 in evaluate File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/data/kernel_tests/test_base.py", line 229 in assertDatasetProduces File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.py", line 189 in testNonLocalRead File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343 in execute_test_method File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360 in decorated File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/absl_py/absl/testing/parameterized.py", line 314 in bound_param_test File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/unittest/case.py", line 549 in _callTestMethod File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/unittest/case.py", line 591 in run File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/unittest/case.py", line 650 in __call__ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/unittest/suite.py", line 122 in run File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/unittest/suite.py", line 84 in __call__ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/unittest/suite.py", line 122 in run File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/unittest/suite.py", line 84 in __call__ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/unittest/runner.py", line 184 in run File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/unittest/main.py", line 271 in runTests File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/unittest/main.py", line 101 in __init__ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/absl_py/absl/testing/absltest.py", line 2537 in _run_and_get_tests_result File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/absl_py/absl/testing/absltest.py", line 2568 in run_tests File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/absl_py/absl/testing/absltest.py", line 2156 in _run_in_app File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/absl_py/absl/testing/absltest.py", line 2049 in main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51 in g_main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/absl_py/absl/app.py", line 258 in _run_main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/absl_py/absl/app.py", line 312 in run File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60 in main_wrapper File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489 in benchmarks_main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62 in main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56 in main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/eager/test.py", line 25 in main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 167 in test_main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/multi_process_cluster.py", line 165 in test_main File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.runfiles/org_tensorflow/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test.py", line 441 in ================================================================================ ==================== Test output for //tensorflow/python/distribute/failure_handling:failure_handler_test (shard 1 of 8): Running tests under Python 3.10.9: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/python_aarch64-unknown-linux-gnu/bin/python3 [ RUN ] PreemptionCheckpointTest.test_error_propagation INFO:tensorflow:Using local port 40743 I0723 22:16:16.845311 281473640004288 test_util.py:3813] Using local port 40743 INFO:tensorflow:Using local port 44561 I0723 22:16:16.845980 281473640004288 test_util.py:3813] Using local port 44561 INFO:tensorflow:Using local port 45971 I0723 22:16:16.846445 281473640004288 test_util.py:3813] Using local port 45971 INFO:tensorflow:Using local port 42887 I0723 22:16:16.846850 281473640004288 test_util.py:3813] Using local port 42887 INFO:tensorflow:Cluster starting. I0723 22:16:20.948313 281473640004288 failure_handler_test.py:387] Cluster starting. [worker-0]: I0723 22:16:20.995099 281473759148736 multi_process_runner.py:840] Subprocess with PID 2277186 (worker, 0) is now being started. [worker-0]: I0723 22:16:20.995596 281473759148736 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:40743", "localhost:44561", "localhost:45971", "localhost:42887"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I0723 22:16:21.055941 281473759148736 multi_process_runner.py:840] Subprocess with PID 2277431 (worker, 1) is now being started. [worker-1]: I0723 22:16:21.056456 281473759148736 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:40743", "localhost:44561", "localhost:45971", "localhost:42887"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I0723 22:16:21.061052 281473759148736 multi_process_runner.py:840] Subprocess with PID 2277575 (worker, 2) is now being started. [worker-2]: I0723 22:16:21.061540 281473759148736 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:40743", "localhost:44561", "localhost:45971", "localhost:42887"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-0]: 2023-07-23 22:16:21.089670: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:40743 [worker-0]: 2023-07-23 22:16:21.092585: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 6860172233737247427 [worker-0]: 2023-07-23 22:16:21.096836: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-2]: 2023-07-23 22:16:21.104152: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:45971 [worker-1]: 2023-07-23 22:16:21.115986: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:44561 [worker-3]: I0723 22:16:21.139184 281473759148736 multi_process_runner.py:840] Subprocess with PID 2277745 (worker, 3) is now being started. [worker-3]: I0723 22:16:21.139658 281473759148736 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:40743", "localhost:44561", "localhost:45971", "localhost:42887"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: 2023-07-23 22:16:21.146168: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 979412107940720586 [worker-2]: 2023-07-23 22:16:21.147064: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-1]: 2023-07-23 22:16:21.147823: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-07-23 22:16:21.147587: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 7008126041722650435 [worker-3]: 2023-07-23 22:16:21.229486: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:42887 [worker-0]: 2023-07-23 22:16:21.243515: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 16738015459993818318 [worker-3]: 2023-07-23 22:16:21.243731: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0723 22:16:21.245693 281473759148736 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0723 22:16:21.245846 281473759148736 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0723 22:16:21.245928 281473759148736 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0723 22:16:21.245670 281473759148736 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0723 22:16:21.302842 281473759148736 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0723 22:16:21.303467 281473759148736 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40743', 'localhost:44561', 'localhost:45971', 'localhost:42887']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0723 22:16:21.303695 281473759148736 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40743', 'localhost:44561', 'localhost:45971', 'localhost:42887']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0723 22:16:21.307236 281473759148736 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0723 22:16:21.307725 281473759148736 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40743', 'localhost:44561', 'localhost:45971', 'localhost:42887']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0723 22:16:21.307949 281473759148736 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40743', 'localhost:44561', 'localhost:45971', 'localhost:42887']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0723 22:16:21.320726 281473759148736 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0723 22:16:21.321307 281473759148736 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40743', 'localhost:44561', 'localhost:45971', 'localhost:42887']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0723 22:16:21.321536 281473759148736 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40743', 'localhost:44561', 'localhost:45971', 'localhost:42887']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0723 22:16:21.321943 281473759148736 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0723 22:16:21.322436 281473759148736 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40743', 'localhost:44561', 'localhost:45971', 'localhost:42887']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0723 22:16:21.322662 281473759148736 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40743', 'localhost:44561', 'localhost:45971', 'localhost:42887']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0723 22:16:21.351986 281473759148736 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I0723 22:16:21.352974 281473759148736 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0723 22:16:21.353291 281473759148736 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0723 22:16:21.353617 281473759148736 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-3]: I0723 22:16:21.352023 281473759148736 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: I0723 22:16:21.353822 281473759148736 failure_handler_test.py:197] Start training at 0 [worker-1]: I0723 22:16:21.352092 281473759148736 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I0723 22:16:21.353223 281473759148736 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0723 22:16:21.353735 281473759148736 failure_handling.py:674] Start watcher for local signal. [worker-1]: I0723 22:16:21.353486 281473759148736 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: I0723 22:16:21.354012 281473759148736 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: Instructions for updating: [worker-0]: I0723 22:16:21.350630 281473759148736 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: W0723 22:16:21.353815 281473759148736 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-1]: Instructions for updating: [worker-3]: W0723 22:16:21.354344 281473759148736 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: I0723 22:16:21.359548 281473759148736 failure_handling.py:674] Start watcher for local signal. [worker-3]: Instructions for updating: [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: I0723 22:16:21.359928 281473759148736 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: INFO:tensorflow:Start training at 0 [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: INFO:tensorflow:Start training at 0 [worker-1]: I0723 22:16:21.354020 281473759148736 failure_handler_test.py:197] Start training at 0 [worker-0]: Instructions for updating: [worker-3]: I0723 22:16:21.354552 281473759148736 failure_handler_test.py:197] Start training at 0 [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0723 22:16:21.360298 281473759148736 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0723 22:16:21.360513 281473759148736 failure_handler_test.py:197] Start training at 0 [worker-2]: INFO:tensorflow:Error reported to Coordinator: in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: Traceback (most recent call last): [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/training/coordinator.py", line 293, in stop_on_exception [worker-2]: yield [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 387, in run [worker-2]: self.main_result = self.main_fn(*self.main_args, **self.main_kwargs) [worker-0]: 2023-07-23 22:16:21.562016: E tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:990] /job:worker/replica:0/task:2 has been set to ERROR in coordination service: RESOURCE_EXHAUSTED: in user code: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/autograph/impl/api.py", line 693, in wrapper [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: raise e.ag_error_metadata.to_exception(e) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: tensorflow.python.framework.errors_impl.ResourceExhaustedError: in user code: [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-0]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: ResourceExhaustedError: Running out of resources [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-0]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-2]: ResourceExhaustedError: Running out of resources [worker-0]: 2023-07-23 22:16:21.562763: E tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:747] Coordination agent is set to ERROR: RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-2]: [worker-0]: [worker-2]: I0723 22:16:21.557793 281447789162976 coordinator.py:213] Error reported to Coordinator: in user code: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-0]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-0]: ResourceExhaustedError: Running out of resources [worker-2]: Traceback (most recent call last): [worker-0]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/training/coordinator.py", line 293, in stop_on_exception [worker-0]: 2023-07-23 22:16:21.562807: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-2]: yield [worker-0]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 387, in run [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: self.main_result = self.main_fn(*self.main_args, **self.main_kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/autograph/impl/api.py", line 693, in wrapper [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-2]: raise e.ag_error_metadata.to_exception(e) [worker-0]: [worker-2]: tensorflow.python.framework.errors_impl.ResourceExhaustedError: in user code: [worker-0]: ResourceExhaustedError: Running out of resources [worker-2]: [worker-0]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: [worker-2]: INFO:tensorflow:Propagating error to cluster: ResourceExhaustedError(): in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: [worker-2]: I0723 22:16:21.560131 281473759148736 failure_handling.py:918] Propagating error to cluster: ResourceExhaustedError(): in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-2]: [worker-2]: 2023-07-23 22:16:21.560455: E tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:747] Coordination agent is set to ERROR: RESOURCE_EXHAUSTED: in user code: [worker-3]: 2023-07-23 22:16:21.576652: E tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:747] Coordination agent is set to ERROR: RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-2]: [worker-3]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-3]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-3]: [worker-2]: ResourceExhaustedError: Running out of resources [worker-3]: ResourceExhaustedError: Running out of resources [worker-2]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-3]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-2]: 2023-07-23 22:16:21.560578: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort RESOURCE_EXHAUSTED: in user code: [worker-2]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-3]: 2023-07-23 22:16:21.576717: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-3]: [worker-2]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: ResourceExhaustedError: Running out of resources [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-1]: 2023-07-23 22:16:21.564083: E tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:747] Coordination agent is set to ERROR: RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: 2023-07-23 22:16:21.560605: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:434] Reporting error to coordination service: RESOURCE_EXHAUSTED: in user code: [worker-1]: [worker-3]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-3]: ResourceExhaustedError: Running out of resources [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: raise errors_impl.ResourceExhaustedError( [worker-1]: [worker-1]: ResourceExhaustedError: Running out of resources [worker-2]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-1]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-1]: 2023-07-23 22:16:21.564134: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort RESOURCE_EXHAUSTED: Error reported from /job:worker/task:2: in user code: [worker-1]: [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-2]: raise errors_impl.ResourceExhaustedError( [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-2]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-2]: ResourceExhaustedError: Running out of resources [worker-1]: raise errors_impl.ResourceExhaustedError( [worker-2]: [worker-1]: [worker-1]: ResourceExhaustedError: Running out of resources [worker-1]: [type.googleapis.com/tensorflow.CoordinationServiceError='\x18\x01\"\n\n\x06worker\x10\x02'] [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:21.574866 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:21.586632 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:21.598729 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-0]: [worker-0]: Detected at node CollectiveReduceV2 defined at (most recent call last): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: test_util.main() [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: test_util.main() [worker-0]: [worker-0]: File "", line 1, in [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: test_util.main() [worker-0]: [worker-0]: File "", line 1, in [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: code = _serve_one(child_r, fds, [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: test_util.main() [worker-0]: [worker-0]: File "", line 1, in [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: code = _serve_one(child_r, fds, [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-0]: code = spawn._main(child_r, parent_sentinel) [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: test_util.main() [worker-0]: [worker-0]: File "", line 1, in [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: code = _serve_one(child_r, fds, [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-0]: code = spawn._main(child_r, parent_sentinel) [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-0]: return self._bootstrap(parent_sentinel) [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: test_util.main() [worker-0]: [worker-0]: File "", line 1, in [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: code = _serve_one(child_r, fds, [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-0]: code = spawn._main(child_r, parent_sentinel) [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-0]: return self._bootstrap(parent_sentinel) [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-0]: self.run() [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: test_util.main() [worker-0]: [worker-0]: File "", line 1, in [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: code = _serve_one(child_r, fds, [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-0]: code = spawn._main(child_r, parent_sentinel) [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-0]: return self._bootstrap(parent_sentinel) [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-0]: self.run() [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: test_util.main() [worker-0]: [worker-0]: File "", line 1, in [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: code = _serve_one(child_r, fds, [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-0]: code = spawn._main(child_r, parent_sentinel) [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-0]: return self._bootstrap(parent_sentinel) [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-0]: self.run() [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-0]: self.worker_fn( [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: Exception in thread PeerTerminationWatcher-1: [worker-1]: Traceback (most recent call last): [worker-0]: test_util.main() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-0]: [worker-0]: File "", line 1, in [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: code = _serve_one(child_r, fds, [worker-1]: INFO:tensorflow:Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-0]: [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: Detected at node CollectiveReduceV2 defined at (most recent call last): [worker-0]: code = spawn._main(child_r, parent_sentinel) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-1]: test_util.main() [worker-0]: return self._bootstrap(parent_sentinel) [worker-1]: [worker-0]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-1]: test_util.main() [worker-0]: self.run() [worker-1]: [worker-0]: [worker-1]: File "", line 1, in [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-1]: [worker-0]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: [worker-1]: test_util.main() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-1]: [worker-0]: self.worker_fn( [worker-1]: File "", line 1, in [worker-0]: [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: preemption_handler.run(distributed_train_step, epoch, step) [worker-1]: code = _serve_one(child_r, fds, [worker-0]: [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: test_util.main() [worker-1]: test_util.main() [worker-0]: [worker-1]: [worker-0]: File "", line 1, in [worker-1]: File "", line 1, in [worker-0]: [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: code = _serve_one(child_r, fds, [worker-1]: code = _serve_one(child_r, fds, [worker-0]: [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-0]: code = spawn._main(child_r, parent_sentinel) [worker-1]: code = spawn._main(child_r, parent_sentinel) [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: [worker-1]: test_util.main() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-1]: [worker-3]: INFO:tensorflow:Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-1]: File "", line 1, in [worker-0]: return self._bootstrap(parent_sentinel) [worker-1]: [worker-3]: [worker-3]: Detected at node CollectiveReduceV2 defined at (most recent call last): [worker-0]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: code = _serve_one(child_r, fds, [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-3]: test_util.main() [worker-0]: self.run() [worker-1]: [worker-0]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-3]: test_util.main() [worker-1]: code = spawn._main(child_r, parent_sentinel) [worker-0]: self._target(*self._args, **self._kwargs) [worker-1]: [worker-3]: [worker-0]: [worker-3]: File "", line 1, in [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-1]: return self._bootstrap(parent_sentinel) [worker-3]: [worker-0]: self.worker_fn( [worker-1]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: [worker-3]: test_util.main() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-3]: [worker-1]: test_util.main() [worker-0]: preemption_handler.run(distributed_train_step, epoch, step) [worker-3]: File "", line 1, in [worker-1]: [worker-0]: [worker-3]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-1]: File "", line 1, in [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: strategy.run(train_step) [worker-1]: [worker-3]: code = _serve_one(child_r, fds, [worker-0]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-3]: [worker-1]: code = _serve_one(child_r, fds, [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: [worker-3]: test_util.main() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-3]: [worker-1]: code = spawn._main(child_r, parent_sentinel) [worker-0]: test_util.main() [worker-3]: File "", line 1, in [worker-1]: [worker-0]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-0]: File "", line 1, in [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-1]: return self._bootstrap(parent_sentinel) [worker-0]: [worker-3]: code = _serve_one(child_r, fds, [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-1]: [worker-3]: [worker-0]: code = _serve_one(child_r, fds, [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: self.run() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: code = spawn._main(child_r, parent_sentinel) [worker-1]: [worker-0]: code = spawn._main(child_r, parent_sentinel) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: [worker-1]: test_util.main() [worker-3]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-1]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: return self._bootstrap(parent_sentinel) [worker-1]: File "", line 1, in [worker-3]: test_util.main() [worker-0]: [worker-1]: [worker-3]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-3]: File "", line 1, in [worker-0]: self.run() [worker-1]: code = _serve_one(child_r, fds, [worker-3]: [worker-0]: [worker-1]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: code = _serve_one(child_r, fds, [worker-0]: self._target(*self._args, **self._kwargs) [worker-1]: code = spawn._main(child_r, parent_sentinel) [worker-3]: [worker-1]: [worker-0]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: return self._bootstrap(parent_sentinel) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-1]: [worker-3]: code = spawn._main(child_r, parent_sentinel) [worker-0]: self.worker_fn( [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-1]: self.run() [worker-3]: [worker-0]: [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-0]: preemption_handler.run(distributed_train_step, epoch, step) [worker-1]: self._target(*self._args, **self._kwargs) [worker-3]: return self._bootstrap(parent_sentinel) [worker-0]: [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-1]: [worker-3]: test_util.main() [worker-0]: strategy.run(train_step) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-3]: [worker-0]: [worker-1]: test_util.main() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-3]: File "", line 1, in [worker-1]: [worker-3]: [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: File "", line 1, in [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: [worker-1]: [worker-3]: code = _serve_one(child_r, fds, [worker-0]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-3]: [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-3]: code = spawn._main(child_r, parent_sentinel) [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-3]: return self._bootstrap(parent_sentinel) [worker-0]: [worker-3]: [worker-0]: ResourceExhaustedError: Running out of resources [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-0]: The error could be from a previous operation. Restart your program to reset. [worker-3]: self.run() [worker-0]: [[{{node CollectiveReduceV2}}]] [worker-3]: [worker-0]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: [Op:__inference_train_step_40] [worker-3]: test_util.main() [worker-0]: I0723 22:16:21.648731 281473759148736 failure_handling.py:918] Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-3]: [worker-0]: [worker-3]: File "", line 1, in [worker-0]: Detected at node CollectiveReduceV2 defined at (most recent call last): [worker-3]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: test_util.main() [worker-0]: [worker-3]: code = _serve_one(child_r, fds, [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-3]: [worker-0]: test_util.main() [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-0]: [worker-3]: code = spawn._main(child_r, parent_sentinel) [worker-0]: File "", line 1, in [worker-3]: [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-3]: return self._bootstrap(parent_sentinel) [worker-0]: test_util.main() [worker-3]: [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-0]: File "", line 1, in [worker-3]: self.run() [worker-0]: [worker-3]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-1]: code = _serve_one(child_r, fds, [worker-0]: code = _serve_one(child_r, fds, [worker-3]: self._target(*self._args, **self._kwargs) [worker-1]: [worker-0]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: code = spawn._main(child_r, parent_sentinel) [worker-0]: test_util.main() [worker-3]: test_util.main() [worker-0]: [worker-1]: [worker-3]: [worker-0]: File "", line 1, in [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-3]: File "", line 1, in [worker-0]: [worker-3]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-1]: return self._bootstrap(parent_sentinel) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: code = _serve_one(child_r, fds, [worker-1]: [worker-3]: code = _serve_one(child_r, fds, [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-0]: [worker-3]: [worker-1]: self.run() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: [worker-3]: code = spawn._main(child_r, parent_sentinel) [worker-0]: code = spawn._main(child_r, parent_sentinel) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-3]: [worker-0]: [worker-1]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: [worker-0]: test_util.main() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-3]: return self._bootstrap(parent_sentinel) [worker-0]: [worker-1]: self.worker_fn( [worker-3]: [worker-0]: File "", line 1, in [worker-1]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-0]: [worker-3]: self.run() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-3]: [worker-1]: test_util.main() [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-1]: [worker-3]: self._target(*self._args, **self._kwargs) [worker-0]: code = _serve_one(child_r, fds, [worker-1]: File "", line 1, in [worker-3]: [worker-0]: [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: code = spawn._main(child_r, parent_sentinel) [worker-3]: self.worker_fn( [worker-1]: code = _serve_one(child_r, fds, [worker-0]: [worker-3]: [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: return self._bootstrap(parent_sentinel) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: test_util.main() [worker-0]: [worker-1]: code = spawn._main(child_r, parent_sentinel) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: [worker-3]: [worker-0]: test_util.main() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-3]: File "", line 1, in [worker-1]: return self._bootstrap(parent_sentinel) [worker-3]: [worker-1]: [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-0]: File "", line 1, in [worker-3]: code = _serve_one(child_r, fds, [worker-1]: self.run() [worker-0]: [worker-3]: [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-0]: code = _serve_one(child_r, fds, [worker-1]: self._target(*self._args, **self._kwargs) [worker-0]: [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-0]: code = spawn._main(child_r, parent_sentinel) [worker-1]: self.worker_fn( [worker-0]: [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-0]: return self._bootstrap(parent_sentinel) [worker-1]: preemption_handler.run(distributed_train_step, epoch, step) [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-1]: [worker-0]: self.run() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: [worker-1]: test_util.main() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: [worker-0]: test_util.main() [worker-1]: File "", line 1, in [worker-0]: [worker-1]: [worker-0]: File "", line 1, in [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: [worker-1]: code = _serve_one(child_r, fds, [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-1]: [worker-0]: code = _serve_one(child_r, fds, [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-0]: [worker-1]: code = spawn._main(child_r, parent_sentinel) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: [worker-0]: code = spawn._main(child_r, parent_sentinel) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-0]: [worker-1]: return self._bootstrap(parent_sentinel) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-0]: return self._bootstrap(parent_sentinel) [worker-1]: [worker-0]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-1]: self.run() [worker-0]: self.run() [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-0]: [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-0]: [worker-1]: self.worker_fn( [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: [worker-0]: test_util.main() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-0]: [worker-1]: preemption_handler.run(distributed_train_step, epoch, step) [worker-0]: File "", line 1, in [worker-1]: [worker-0]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-1]: strategy.run(train_step) [worker-0]: code = _serve_one(child_r, fds, [worker-1]: [worker-0]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: test_util.main() [worker-0]: code = spawn._main(child_r, parent_sentinel) [worker-1]: [worker-0]: [worker-1]: File "", line 1, in [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-1]: [worker-0]: return self._bootstrap(parent_sentinel) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: [worker-1]: code = _serve_one(child_r, fds, [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-1]: [worker-0]: self.run() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-0]: [worker-1]: code = spawn._main(child_r, parent_sentinel) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-1]: [worker-0]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-0]: [worker-1]: return self._bootstrap(parent_sentinel) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-1]: [worker-0]: self.worker_fn( [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-0]: [worker-1]: self.run() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: [worker-0]: test_util.main() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-0]: [worker-1]: self._target(*self._args, **self._kwargs) [worker-0]: File "", line 1, in [worker-0]: [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: code = _serve_one(child_r, fds, [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-0]: [worker-1]: self.worker_fn( [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: [worker-0]: code = spawn._main(child_r, parent_sentinel) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-0]: [worker-3]: code = spawn._main(child_r, parent_sentinel) [worker-1]: preemption_handler.run(distributed_train_step, epoch, step) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-3]: [worker-0]: return self._bootstrap(parent_sentinel) [worker-1]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-0]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-3]: return self._bootstrap(parent_sentinel) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-1]: strategy.run(train_step) [worker-3]: [worker-0]: self.run() [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-1]: [worker-0]: [worker-3]: self.run() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-3]: [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-1]: [worker-0]: [worker-3]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-0]: self.worker_fn( [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-0]: preemption_handler.run(distributed_train_step, epoch, step) [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: test_util.main() [worker-0]: [worker-3]: [worker-0]: File "", line 1, in [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-0]: [worker-3]: self.worker_fn( [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-3]: [worker-0]: code = _serve_one(child_r, fds, [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-0]: [worker-3]: preemption_handler.run(distributed_train_step, epoch, step) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: [worker-0]: code = spawn._main(child_r, parent_sentinel) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-0]: return self._bootstrap(parent_sentinel) [worker-3]: test_util.main() [worker-0]: [worker-3]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-1]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-3]: File "", line 1, in [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-3]: code = _serve_one(child_r, fds, [worker-0]: self.run() [worker-1]: [worker-3]: [worker-0]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: code = spawn._main(child_r, parent_sentinel) [worker-0]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-3]: [worker-0]: [worker-1]: raise errors_impl.ResourceExhaustedError( [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-1]: [worker-3]: return self._bootstrap(parent_sentinel) [worker-0]: self.worker_fn( [worker-1]: ResourceExhaustedError: Running out of resources [worker-3]: [worker-0]: [worker-1]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-1]: The error could be from a previous operation. Restart your program to reset. [worker-3]: self.run() [worker-0]: preemption_handler.run(distributed_train_step, epoch, step) [worker-1]: [[{{node CollectiveReduceV2}}]] [worker-0]: [worker-3]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-1]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-0]: strategy.run(train_step) [worker-3]: self._target(*self._args, **self._kwargs) [worker-1]: [Op:__inference_train_step_38] [worker-0]: [worker-3]: [worker-1]: I0723 22:16:21.687922 281473759148736 failure_handling.py:918] Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-1]: [worker-0]: test_util.main() [worker-3]: self.worker_fn( [worker-1]: Detected at node CollectiveReduceV2 defined at (most recent call last): [worker-0]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: File "", line 1, in [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-1]: test_util.main() [worker-0]: [worker-3]: preemption_handler.run(distributed_train_step, epoch, step) [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: code = _serve_one(child_r, fds, [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-1]: test_util.main() [worker-0]: [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: strategy.run(train_step) [worker-1]: File "", line 1, in [worker-0]: code = spawn._main(child_r, parent_sentinel) [worker-1]: [worker-3]: [worker-0]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-1]: test_util.main() [worker-3]: test_util.main() [worker-0]: return self._bootstrap(parent_sentinel) [worker-1]: [worker-3]: [worker-0]: [worker-1]: File "", line 1, in [worker-3]: File "", line 1, in [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-1]: [worker-0]: self.run() [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-1]: code = _serve_one(child_r, fds, [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-3]: code = _serve_one(child_r, fds, [worker-1]: [worker-0]: self._target(*self._args, **self._kwargs) [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: test_util.main() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-3]: code = spawn._main(child_r, parent_sentinel) [worker-1]: [worker-0]: self.worker_fn( [worker-3]: [worker-1]: File "", line 1, in [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-3]: return self._bootstrap(parent_sentinel) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-0]: preemption_handler.run(distributed_train_step, epoch, step) [worker-3]: [worker-1]: code = _serve_one(child_r, fds, [worker-0]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-1]: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-3]: self.run() [worker-0]: strategy.run(train_step) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: [worker-0]: [worker-1]: code = spawn._main(child_r, parent_sentinel) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-1]: [worker-3]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-1]: test_util.main() [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: self.worker_fn( [worker-1]: [worker-0]: [worker-3]: [worker-1]: File "", line 1, in [worker-0]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-1]: [worker-0]: [worker-3]: preemption_handler.run(distributed_train_step, epoch, step) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-3]: [worker-1]: code = _serve_one(child_r, fds, [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-1]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-0]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: strategy.run(train_step) [worker-1]: code = spawn._main(child_r, parent_sentinel) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-3]: [worker-1]: [worker-0]: raise errors_impl.ResourceExhaustedError( [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-0]: [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-0]: ResourceExhaustedError: Running out of resources [worker-0]: [worker-1]: return self._bootstrap(parent_sentinel) [worker-3]: [worker-0]: The error could be from a previous operation. Restart your program to reset. [worker-1]: [worker-3]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-0]: [[{{node CollectiveReduceV2}}]] [worker-3]: [worker-1]: test_util.main() [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-0]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-1]: [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-1]: File "", line 1, in [worker-0]: [Op:__inference_train_step_40] [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-1]: [worker-0]: INFO:tensorflow:Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. [worker-3]: raise errors_impl.ResourceExhaustedError( [worker-0]: I0723 22:16:21.649307 281473759148736 failure_handling.py:922] Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-3]: [worker-1]: code = _serve_one(child_r, fds, [worker-3]: ResourceExhaustedError: Running out of resources [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: The error could be from a previous operation. Restart your program to reset. [worker-1]: code = spawn._main(child_r, parent_sentinel) [worker-3]: [[{{node CollectiveReduceV2}}]] [worker-1]: [worker-3]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-3]: [Op:__inference_train_step_38] [worker-1]: return self._bootstrap(parent_sentinel) [worker-3]: Exception in thread PeerTerminationWatcher-3: [worker-1]: [worker-3]: Traceback (most recent call last): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/threading.py", line 1016, in _bootstrap_inner [worker-1]: self.run() [worker-3]: self.run() [worker-1]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/threading.py", line 953, in run [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-3]: self._target(*self._args, **self._kwargs) [worker-1]: test_util.main() [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 1229, in _watch_step_to_save_key [worker-1]: [worker-3]: step_value = context.context().get_config_key_value(_INITIAL_RUN_COUNT_KEY) [worker-1]: File "", line 1, in [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 806, in get_config_key_value [worker-1]: [worker-3]: pywrap_tfe.TFE_GetConfigKeyValue(self._context_handle, key, [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-3]: tensorflow.python.framework.errors_impl.UnavailableError: Socket closed [worker-1]: code = _serve_one(child_r, fds, [worker-3]: Additional GRPC error information from remote target /job:worker/replica:0/task:0 while calling /tensorflow.CoordinationService/GetKeyValue: [worker-1]: [worker-3]: :{"created":"@1690150581.684010544","description":"Error received from peer ipv6:[::1]:40743","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket closed","grpc_status":14} [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: I0723 22:16:21.684742 281473759148736 failure_handling.py:918] Propagating error to cluster: ResourceExhaustedError(): Graph execution error: [worker-1]: code = spawn._main(child_r, parent_sentinel) [worker-3]: [worker-1]: [worker-3]: Detected at node CollectiveReduceV2 defined at (most recent call last): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: return self._bootstrap(parent_sentinel) [worker-3]: test_util.main() [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: self.run() [worker-3]: test_util.main() [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-3]: File "", line 1, in [worker-1]: self._target(*self._args, **self._kwargs) [worker-3]: [worker-1]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-3]: test_util.main() [worker-1]: test_util.main() [worker-3]: [worker-1]: [worker-1]: File "", line 1, in [worker-3]: File "", line 1, in [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-1]: code = _serve_one(child_r, fds, [worker-3]: code = _serve_one(child_r, fds, [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: code = spawn._main(child_r, parent_sentinel) [worker-3]: test_util.main() [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-3]: File "", line 1, in [worker-1]: return self._bootstrap(parent_sentinel) [worker-3]: [worker-1]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-3]: code = _serve_one(child_r, fds, [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-3]: [worker-1]: self.run() [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: [worker-3]: code = spawn._main(child_r, parent_sentinel) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-3]: [worker-1]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: [worker-3]: test_util.main() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-3]: [worker-1]: self.worker_fn( [worker-3]: File "", line 1, in [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-1]: test_util.main() [worker-3]: code = _serve_one(child_r, fds, [worker-1]: [worker-3]: [worker-1]: File "", line 1, in [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: [worker-3]: code = spawn._main(child_r, parent_sentinel) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-1]: code = _serve_one(child_r, fds, [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: code = spawn._main(child_r, parent_sentinel) [worker-3]: [worker-1]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-3]: return self._bootstrap(parent_sentinel) [worker-1]: return self._bootstrap(parent_sentinel) [worker-3]: [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: self.run() [worker-1]: [worker-3]: test_util.main() [worker-3]: [worker-3]: File "", line 1, in [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-3]: [worker-1]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-1]: [worker-3]: code = _serve_one(child_r, fds, [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-3]: [worker-1]: self.worker_fn( [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: [worker-3]: code = spawn._main(child_r, parent_sentinel) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-3]: [worker-1]: preemption_handler.run(distributed_train_step, epoch, step) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-1]: [worker-3]: return self._bootstrap(parent_sentinel) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: test_util.main() [worker-3]: [worker-1]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-1]: File "", line 1, in [worker-3]: self.run() [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: code = _serve_one(child_r, fds, [worker-3]: test_util.main() [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: File "", line 1, in [worker-1]: code = spawn._main(child_r, parent_sentinel) [worker-3]: [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-1]: return self._bootstrap(parent_sentinel) [worker-3]: code = _serve_one(child_r, fds, [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: self.run() [worker-3]: code = spawn._main(child_r, parent_sentinel) [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-1]: self._target(*self._args, **self._kwargs) [worker-3]: return self._bootstrap(parent_sentinel) [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-1]: self.worker_fn( [worker-3]: self.run() [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-1]: preemption_handler.run(distributed_train_step, epoch, step) [worker-1]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-3]: self._target(*self._args, **self._kwargs) [worker-1]: strategy.run(train_step) [worker-3]: [worker-1]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: test_util.main() [worker-1]: [worker-3]: test_util.main() [worker-1]: File "", line 1, in [worker-3]: [worker-3]: File "", line 1, in [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-1]: code = _serve_one(child_r, fds, [worker-3]: code = _serve_one(child_r, fds, [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: code = spawn._main(child_r, parent_sentinel) [worker-3]: code = spawn._main(child_r, parent_sentinel) [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-1]: return self._bootstrap(parent_sentinel) [worker-3]: return self._bootstrap(parent_sentinel) [worker-1]: [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-1]: self.run() [worker-1]: [worker-3]: self.run() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-3]: [worker-1]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-1]: [worker-3]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-3]: [worker-1]: self.worker_fn( [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-1]: [worker-3]: self.worker_fn( [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-3]: [worker-1]: preemption_handler.run(distributed_train_step, epoch, step) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: [worker-3]: test_util.main() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-3]: [worker-1]: strategy.run(train_step) [worker-3]: File "", line 1, in [worker-1]: [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-3]: code = _serve_one(child_r, fds, [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-1]: [worker-3]: code = spawn._main(child_r, parent_sentinel) [worker-1]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-3]: [worker-1]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-3]: return self._bootstrap(parent_sentinel) [worker-1]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-1]: raise errors_impl.ResourceExhaustedError( [worker-3]: self.run() [worker-1]: [worker-3]: [worker-1]: ResourceExhaustedError: Running out of resources [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-1]: [worker-3]: self._target(*self._args, **self._kwargs) [worker-1]: The error could be from a previous operation. Restart your program to reset. [worker-3]: [worker-1]: [[{{node CollectiveReduceV2}}]] [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-1]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-3]: self.worker_fn( [worker-1]: [Op:__inference_train_step_38] [worker-3]: [worker-1]: INFO:tensorflow:Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-1]: I0723 22:16:21.688449 281473759148736 failure_handling.py:922] Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. [worker-3]: preemption_handler.run(distributed_train_step, epoch, step) [worker-1]: self.run() [worker-3]: [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/threading.py", line 953, in run [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-1]: self._target(*self._args, **self._kwargs) [worker-3]: test_util.main() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 1229, in _watch_step_to_save_key [worker-3]: [worker-1]: step_value = context.context().get_config_key_value(_INITIAL_RUN_COUNT_KEY) [worker-3]: File "", line 1, in [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 806, in get_config_key_value [worker-3]: [worker-1]: pywrap_tfe.TFE_GetConfigKeyValue(self._context_handle, key, [worker-1]: tensorflow.python.framework.errors_impl.UnavailableError: Socket closed [worker-1]: Additional GRPC error information from remote target /job:worker/replica:0/task:0 while calling /tensorflow.CoordinationService/GetKeyValue: [worker-1]: :{"created":"@1690150581.683579344","description":"Error received from peer ipv6:[::1]:40743","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket closed","grpc_status":14} [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-3]: code = _serve_one(child_r, fds, [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: code = spawn._main(child_r, parent_sentinel) [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-3]: return self._bootstrap(parent_sentinel) [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-3]: self.run() [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-3]: self.worker_fn( [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-3]: preemption_handler.run(distributed_train_step, epoch, step) [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-3]: strategy.run(train_step) [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 558, in [worker-3]: test_util.main() [worker-3]: [worker-3]: File "", line 1, in [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 274, in main [worker-3]: code = _serve_one(child_r, fds, [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/forkserver.py", line 313, in _serve_one [worker-3]: code = spawn._main(child_r, parent_sentinel) [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/spawn.py", line 129, in _main [worker-3]: return self._bootstrap(parent_sentinel) [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [worker-3]: self.run() [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/python_aarch64-unknown-linux-gnu/lib/python3.10/multiprocessing/process.py", line 108, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 367, in assert_raise_error [worker-3]: self.worker_fn( [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 229, in worker_fn [worker-3]: preemption_handler.run(distributed_train_step, epoch, step) [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 192, in distributed_train_step [worker-3]: strategy.run(train_step) [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: [worker-3]: Collective ops is aborted by: Error reported from /job:worker/task:2: in user code: [worker-3]: [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/mirrored_run.py", line 79, in wrapped_fn * [worker-3]: return call_for_each_replica(strategy, fn.python_function, args, kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 187, in train_step * [worker-3]: raise errors_impl.ResourceExhaustedError( [worker-3]: [worker-3]: ResourceExhaustedError: Running out of resources [worker-3]: [worker-3]: The error could be from a previous operation. Restart your program to reset. [worker-3]: [[{{node CollectiveReduceV2}}]] [worker-3]: Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [worker-3]: [Op:__inference_train_step_38] [worker-3]: INFO:tensorflow:Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. [worker-3]: I0723 22:16:21.688233 281473759148736 failure_handling.py:922] Ignoring error during error propagation: FailedPreconditionError():Coordination service agent is already in error state. I0723 22:16:22.006731 281473640004288 multi_process_runner.py:646] worker-0 exit code: 0 I0723 22:16:22.007028 281473640004288 multi_process_runner.py:646] worker-1 exit code: 0 I0723 22:16:22.007205 281473640004288 multi_process_runner.py:646] worker-2 exit code: 0 I0723 22:16:22.007370 281473640004288 multi_process_runner.py:646] worker-3 exit code: 0 I0723 22:16:22.011214 281473640004288 multi_process_runner.py:662] Joining log reading threads. I0723 22:16:22.011512 281473640004288 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_error_propagation): 5.5s I0723 22:16:22.344718 281473640004288 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_error_propagation): 5.5s [ OK ] PreemptionCheckpointTest.test_error_propagation [ RUN ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice INFO:tensorflow:Start watcher for local signal. I0723 22:16:22.438380 281473640004288 failure_handling.py:674] Start watcher for local signal. INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. I0723 22:16:22.438755 281473640004288 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. W0723 22:16:22.439083 281473640004288 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. INFO:tensorflow:Start training at 0 I0723 22:16:22.439303 281473640004288 failure_handler_test.py:197] Start training at 0 WARNING:tensorflow:5 out of the last 5 calls to .distributed_train_step..train_step at 0xffff03b3f7f0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W0723 22:16:22.816493 281473640004288 polymorphic_function.py:156] 5 out of the last 5 calls to .distributed_train_step..train_step at 0xffff03b3f7f0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. WARNING:tensorflow:6 out of the last 6 calls to .distributed_train_step..train_step at 0xffff03b3f7f0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W0723 22:16:22.830601 281473640004288 polymorphic_function.py:156] 6 out of the last 6 calls to .distributed_train_step..train_step at 0xffff03b3f7f0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. INFO:tensorflow:epoch 0 finished I0723 22:16:22.962033 281473640004288 failure_handler_test.py:195] epoch 0 finished INFO:tensorflow:epoch 1 finished I0723 22:16:23.192802 281473640004288 failure_handler_test.py:195] epoch 1 finished INFO:tensorflow:epoch 2 finished I0723 22:16:23.417182 281473640004288 failure_handler_test.py:195] epoch 2 finished INFO:tensorflow:epoch 3 finished I0723 22:16:23.645044 281473640004288 failure_handler_test.py:195] epoch 3 finished INFO:tensorflow:epoch 4 finished I0723 22:16:23.940021 281473640004288 failure_handler_test.py:195] epoch 4 finished INFO:tensorflow:epoch 5 finished I0723 22:16:24.169224 281473640004288 failure_handler_test.py:195] epoch 5 finished INFO:tensorflow:sending sigterm I0723 22:16:24.383785 281470735151584 failure_handler_test.py:467] sending sigterm INFO:tensorflow:Member single_worker has received termination notice. I0723 22:16:24.384963 281473640004288 failure_handling.py:701] Member single_worker has received termination notice. INFO:tensorflow:Termination caught in main thread on preempted worker I0723 22:16:24.385942 281473640004288 failure_handling.py:1159] Termination caught in main thread on preempted worker INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. I0723 22:16:24.400826 281473640004288 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b904upa10yp/tmpgpiw6u4k/fh_ckpt I0723 22:16:24.446954 281473640004288 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b904upa10yp/tmpgpiw6u4k/fh_ckpt INFO:tensorflow:Continue training for the grace period. I0723 22:16:24.447300 281473640004288 failure_handling.py:1134] Continue training for the grace period. INFO:tensorflow:epoch 6 finished I0723 22:16:24.462605 281473640004288 failure_handler_test.py:195] epoch 6 finished INFO:tensorflow:epoch 7 finished I0723 22:16:24.687870 281473640004288 failure_handler_test.py:195] epoch 7 finished INFO:tensorflow:Training finished. I0723 22:16:24.688458 281473640004288 failure_handler_test.py:245] Training finished. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice): 2.34s I0723 22:16:24.689494 281473640004288 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice): 2.34s [ OK ] PreemptionCheckpointTest.test_grace_period_continue_training_test_inputarg_manager_strategyoption_OneDevice [ RUN ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 40479 I0723 22:16:24.692486 281473640004288 test_util.py:3813] Using local port 40479 INFO:tensorflow:Using local port 40225 I0723 22:16:24.692942 281473640004288 test_util.py:3813] Using local port 40225 INFO:tensorflow:Using local port 35819 I0723 22:16:24.693358 281473640004288 test_util.py:3813] Using local port 35819 INFO:tensorflow:Using local port 37017 I0723 22:16:24.693762 281473640004288 test_util.py:3813] Using local port 37017 INFO:tensorflow:Cluster starting. I0723 22:16:24.805032 281473640004288 failure_handler_test.py:297] Cluster starting. [worker-0]: I0723 22:16:24.858723 281473759148736 multi_process_runner.py:840] Subprocess with PID 2288595 (worker, 0) is now being started. [worker-0]: I0723 22:16:24.859199 281473759148736 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:40479", "localhost:40225", "localhost:35819", "localhost:37017"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I0723 22:16:24.946770 281473759148736 multi_process_runner.py:840] Subprocess with PID 2288612 (worker, 1) is now being started. [worker-1]: I0723 22:16:24.947261 281473759148736 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:40479", "localhost:40225", "localhost:35819", "localhost:37017"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I0723 22:16:24.960979 281473759148736 multi_process_runner.py:840] Subprocess with PID 2288734 (worker, 2) is now being started. [worker-2]: I0723 22:16:24.961466 281473759148736 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:40479", "localhost:40225", "localhost:35819", "localhost:37017"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0723 22:16:24.979555 281473759148736 multi_process_runner.py:840] Subprocess with PID 2288849 (worker, 3) is now being started. [worker-0]: 2023-07-23 22:16:24.997228: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:40479 [worker-0]: 2023-07-23 22:16:25.007152: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 15937675331925395834 [worker-0]: 2023-07-23 22:16:25.007819: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-3]: I0723 22:16:24.980038 281473759148736 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:40479", "localhost:40225", "localhost:35819", "localhost:37017"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-2]: 2023-07-23 22:16:25.013511: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:35819 [worker-3]: 2023-07-23 22:16:25.018943: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:37017 [worker-0]: 2023-07-23 22:16:25.032477: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 7607159683385301820 [worker-2]: 2023-07-23 22:16:25.032694: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-3]: 2023-07-23 22:16:25.045703: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-07-23 22:16:25.045318: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 4426213606466324290 [worker-0]: 2023-07-23 22:16:25.076499: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 8549897635718601692 [worker-1]: 2023-07-23 22:16:25.066901: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:40225 [worker-1]: 2023-07-23 22:16:25.083924: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0723 22:16:25.086791 281473759148736 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0723 22:16:25.086663 281473759148736 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0723 22:16:25.109063 281473759148736 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I0723 22:16:25.097354 281473759148736 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0723 22:16:25.146763 281473759148736 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0723 22:16:25.147268 281473759148736 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40479', 'localhost:40225', 'localhost:35819', 'localhost:37017']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0723 22:16:25.147494 281473759148736 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40479', 'localhost:40225', 'localhost:35819', 'localhost:37017']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0723 22:16:25.174885 281473759148736 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0723 22:16:25.175412 281473759148736 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40479', 'localhost:40225', 'localhost:35819', 'localhost:37017']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0723 22:16:25.175639 281473759148736 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40479', 'localhost:40225', 'localhost:35819', 'localhost:37017']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0723 22:16:25.174754 281473759148736 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0723 22:16:25.175281 281473759148736 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40479', 'localhost:40225', 'localhost:35819', 'localhost:37017']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0723 22:16:25.175509 281473759148736 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40479', 'localhost:40225', 'localhost:35819', 'localhost:37017']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0723 22:16:25.213889 281473759148736 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0723 22:16:25.214730 281473759148736 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40479', 'localhost:40225', 'localhost:35819', 'localhost:37017']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0723 22:16:25.214956 281473759148736 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40479', 'localhost:40225', 'localhost:35819', 'localhost:37017']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0723 22:16:25.297966 281473759148736 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-2]: I0723 22:16:25.299181 281473759148736 failure_handling.py:674] Start watcher for local signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0723 22:16:25.299453 281473759148736 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0723 22:16:25.299787 281473759148736 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0723 22:16:25.299992 281473759148736 failure_handler_test.py:197] Start training at 0 [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0723 22:16:25.315671 281473759148736 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0723 22:16:25.325141 281473759148736 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I0723 22:16:25.325946 281473759148736 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I0723 22:16:25.326229 281473759148736 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: I0723 22:16:25.326900 281473759148736 failure_handling.py:674] Start watcher for local signal. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: Instructions for updating: [worker-0]: I0723 22:16:25.327232 281473759148736 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: I0723 22:16:25.328707 281473759148736 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0723 22:16:25.326560 281473759148736 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-1]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-0]: W0723 22:16:25.327608 281473759148736 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-1]: I0723 22:16:25.326781 281473759148736 failure_handler_test.py:197] Start training at 0 [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0723 22:16:25.327822 281473759148736 failure_handler_test.py:197] Start training at 0 [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I0723 22:16:25.356582 281473759148736 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0723 22:16:25.356943 281473759148736 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0723 22:16:25.357297 281473759148736 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0723 22:16:25.357533 281473759148736 failure_handler_test.py:197] Start training at 0 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:25.531287 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:25.534961 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:25.568223 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:25.668124 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:25.766956 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:25.768651 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:25.771178 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:25.765925 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:25.899742 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:25.929553 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:25.954942 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:26.011472 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:26.161473 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:26.169519 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:26.159484 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:26.171607 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:26.304654 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:26.310041 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:26.310825 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:26.311498 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffb32f9240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0723 22:16:26.407449 281473759148736 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffb32f9240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffb32fd240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffb32f9240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0723 22:16:26.414371 281473759148736 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffb32f9240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0723 22:16:26.408073 281473759148736 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffb32fd240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffb32f9240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0723 22:16:26.416543 281473759148736 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffb32f9240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:26.427357 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:26.419097 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:26.427407 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:26.422554 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffb32fa170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0723 22:16:26.497344 281473759148736 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffb32fa170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffb32fa170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffb32fa170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0723 22:16:26.507245 281473759148736 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffb32fa170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffb32fe170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0723 22:16:26.507527 281473759148736 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffb32fa170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0723 22:16:26.507910 281473759148736 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffb32fe170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:26.518014 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:26.519997 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:26.520523 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:26.525480 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:26.584063 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:26.590406 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:26.591860 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:26.591979 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:26.650712 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:26.659818 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:26.660269 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:26.667337 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:26.819453 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:26.819485 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:26.819656 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:26.877450 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:26.938503 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:26.938527 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:26.938668 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:26.940780 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:27.001947 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:27.002098 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:27.002183 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:27.004305 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:27.064208 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:27.064458 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:27.064245 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:27.066460 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:27.129404 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:27.129464 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:27.129700 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:27.130484 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:27.191005 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:27.191035 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:27.192797 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:27.193988 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I0723 22:16:27.266923 281473759148736 failure_handler_test.py:195] epoch 0 finished [worker-2]: INFO:tensorflow:epoch 0 finished [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I0723 22:16:27.276780 281473759148736 failure_handler_test.py:195] epoch 0 finished [worker-2]: I0723 22:16:27.278074 281473759148736 failure_handler_test.py:195] epoch 0 finished [worker-1]: INFO:tensorflow:epoch 0 finished [worker-1]: I0723 22:16:27.316609 281473759148736 failure_handler_test.py:195] epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:27.321322 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:27.325538 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:27.329644 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:27.325176 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:27.386059 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:27.387372 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:27.387410 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:27.387449 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:27.446251 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:27.446280 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:27.446305 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:27.446911 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:27.505067 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:27.505086 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:27.506616 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:27.506665 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:27.563027 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:27.564167 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:27.564193 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:27.564204 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:27.620501 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:27.620527 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:27.622094 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:27.622505 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:27.683613 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:27.684869 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:27.685807 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:27.687268 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:27.747003 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:27.749093 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:27.748705 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:27.748458 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:27.815517 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:27.817641 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:27.823213 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:27.812661 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:27.883046 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:27.892113 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:27.893543 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:27.896790 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:27.961308 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:27.961409 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:27.963287 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:27.974048 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:28.077753 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:28.069466 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:28.070099 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:28.067804 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:28.247511 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:28.251467 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:28.251340 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:28.273690 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:28.378241 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:28.383740 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:28.406454 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:28.413141 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:28.473496 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:28.478174 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:28.501708 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:28.505375 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 1 finished [worker-1]: I0723 22:16:28.570292 281473759148736 failure_handler_test.py:195] epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0723 22:16:28.573346 281473759148736 failure_handler_test.py:195] epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-2]: I0723 22:16:28.577608 281473759148736 failure_handler_test.py:195] epoch 1 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:28.583401 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I0723 22:16:28.586536 281473759148736 failure_handler_test.py:195] epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:28.587540 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:28.591479 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:28.618006 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:28.710999 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:28.714897 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:28.712660 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:28.714302 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:28.786641 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:28.786652 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:28.787369 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:28.787074 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:28.849682 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:28.849711 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:28.852553 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:28.868363 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:28.941296 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:28.928617 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:sending sigterm I0723 22:16:28.966508 281473640004288 failure_handler_test.py:302] sending sigterm INFO:tensorflow:sigterm sent I0723 22:16:28.966893 281473640004288 failure_handler_test.py:306] sigterm sent [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:28.961560 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:28.981315 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Member 2 has received termination notice. [worker-2]: I0723 22:16:29.061485 281473759148736 failure_handling.py:710] Member 2 has received termination notice. [worker-2]: INFO:tensorflow:Termination caught in main thread on preempted worker [worker-2]: I0723 22:16:29.116806 281473759148736 failure_handling.py:1159] Termination caught in main thread on preempted worker [worker-2]: INFO:tensorflow:RUN_TO_CHECKPOINT set to 36 [worker-2]: I0723 22:16:29.117973 281473759148736 failure_handling.py:1168] RUN_TO_CHECKPOINT set to 36 [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-2]: I0723 22:16:29.118771 281447663399392 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_2 set, preemption awareness acknowledged [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 0 received [worker-1]: I0723 22:16:29.119558 281447948218848 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_1 set, preemption awareness acknowledged [worker-2]: I0723 22:16:29.119910 281473759148736 failure_handling.py:1177] Sigterm acknowledgement from replica 0 received [worker-0]: I0723 22:16:29.119967 281452344111584 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_0 set, preemption awareness acknowledged [worker-3]: I0723 22:16:29.119836 281448896197088 failure_handling.py:1242] PreemptionCheckpointHandler: RECEIVED_SIGNAL_RUN_TO_CHECKPOINT_3 set, preemption awareness acknowledged [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 1 received [worker-2]: I0723 22:16:29.120563 281473759148736 failure_handling.py:1177] Sigterm acknowledgement from replica 1 received [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 2 received [worker-2]: I0723 22:16:29.121295 281473759148736 failure_handling.py:1177] Sigterm acknowledgement from replica 2 received [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Sigterm acknowledgement from replica 3 received [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:29.122217 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:29.121803 281473759148736 failure_handling.py:1177] Sigterm acknowledgement from replica 3 received [worker-0]: I0723 22:16:29.122380 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:29.123077 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:29.131701 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-3]: I0723 22:16:29.267342 281473759148736 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: I0723 22:16:29.297757 281473759148736 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-1]: I0723 22:16:29.309675 281473759148736 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-0]: I0723 22:16:29.285636 281473759148736 failure_handling.py:1063] PreemptionCheckpointHandler: Starting saving a checkpoint. [worker-2]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b904upa10yp/tmpo5f526lm/workertemp_2/fh_ckpt [worker-2]: I0723 22:16:29.342551 281473759148736 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b904upa10yp/tmpo5f526lm/workertemp_2/fh_ckpt [worker-2]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-3]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b904upa10yp/tmpo5f526lm/workertemp_3/fh_ckpt [worker-3]: I0723 22:16:29.354326 281473759148736 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b904upa10yp/tmpo5f526lm/workertemp_3/fh_ckpt [worker-3]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-3]: I0723 22:16:29.357504 281473759148736 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-0]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b904upa10yp/tmpo5f526lm/fh_ckpt [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-3]: I0723 22:16:29.357765 281473759148736 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: I0723 22:16:29.358558 281473759148736 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b904upa10yp/tmpo5f526lm/fh_ckpt [worker-0]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-0]: I0723 22:16:29.361381 281473759148736 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-0]: I0723 22:16:29.361610 281473759148736 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-2]: I0723 22:16:29.346733 281473759148736 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-2]: I0723 22:16:29.347039 281473759148736 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-1]: INFO:tensorflow:Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b904upa10yp/tmpo5f526lm/workertemp_1/fh_ckpt [worker-1]: I0723 22:16:29.381768 281473759148736 failure_handling.py:1078] Checkpoint finished at path /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/_tmp/1974f2236c3f9f2f111c0901320a7b904upa10yp/tmpo5f526lm/workertemp_1/fh_ckpt [worker-1]: INFO:tensorflow:Shut down watcher for peer's termination signal. [worker-1]: I0723 22:16:29.383389 281473759148736 failure_handling.py:771] Shut down watcher for peer's termination signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler: checkpoint saved. Exiting. [worker-1]: I0723 22:16:29.383608 281473759148736 failure_handling.py:1128] PreemptionCheckpointHandler: checkpoint saved. Exiting. INFO:tensorflow:restarting workers I0723 22:16:30.976194 281473640004288 failure_handler_test.py:309] restarting workers INFO:tensorflow:workers restarted I0723 22:16:31.121406 281473640004288 failure_handler_test.py:313] workers restarted [worker-0]: I0723 22:16:31.118254 281473759148736 multi_process_runner.py:840] Subprocess with PID 2304715 (worker, 0) is now being started. [worker-0]: I0723 22:16:31.118741 281473759148736 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:40479", "localhost:40225", "localhost:35819", "localhost:37017"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I0723 22:16:31.160001 281473759148736 multi_process_runner.py:840] Subprocess with PID 2304793 (worker, 1) is now being started. [worker-0]: 2023-07-23 22:16:31.209410: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:40479 [worker-1]: I0723 22:16:31.160521 281473759148736 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:40479", "localhost:40225", "localhost:35819", "localhost:37017"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-3]: I0723 22:16:31.238593 281473759148736 multi_process_runner.py:840] Subprocess with PID 2304898 (worker, 3) is now being started. [worker-3]: I0723 22:16:31.239125 281473759148736 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:40479", "localhost:40225", "localhost:35819", "localhost:37017"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-1]: 2023-07-23 22:16:31.287745: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:40225 [worker-2]: I0723 22:16:31.301376 281473759148736 multi_process_runner.py:840] Subprocess with PID 2304820 (worker, 2) is now being started. [worker-2]: I0723 22:16:31.301926 281473759148736 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:40479", "localhost:40225", "localhost:35819", "localhost:37017"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-0]: 2023-07-23 22:16:31.310164: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 3727110102898704964 [worker-0]: 2023-07-23 22:16:31.310725: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-07-23 22:16:31.311142: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 11311605286944060874 [worker-1]: 2023-07-23 22:16:31.311348: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-3]: 2023-07-23 22:16:31.317999: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:37017 [worker-0]: 2023-07-23 22:16:31.331144: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 1713909215971659153 [worker-3]: 2023-07-23 22:16:31.331383: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-2]: 2023-07-23 22:16:31.338494: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:35819 [worker-0]: 2023-07-23 22:16:31.345447: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 2361000846483008522 [worker-2]: 2023-07-23 22:16:31.345715: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0723 22:16:31.348573 281473759148736 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0723 22:16:31.348612 281473759148736 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0723 22:16:31.348927 281473759148736 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0723 22:16:31.357597 281473759148736 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0723 22:16:31.404523 281473759148736 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0723 22:16:31.405065 281473759148736 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40479', 'localhost:40225', 'localhost:35819', 'localhost:37017']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0723 22:16:31.405624 281473759148736 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-0]: I0723 22:16:31.405292 281473759148736 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40479', 'localhost:40225', 'localhost:35819', 'localhost:37017']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0723 22:16:31.406405 281473759148736 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40479', 'localhost:40225', 'localhost:35819', 'localhost:37017']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0723 22:16:31.406651 281473759148736 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40479', 'localhost:40225', 'localhost:35819', 'localhost:37017']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0723 22:16:31.411041 281473759148736 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0723 22:16:31.411749 281473759148736 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40479', 'localhost:40225', 'localhost:35819', 'localhost:37017']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0723 22:16:31.411990 281473759148736 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40479', 'localhost:40225', 'localhost:35819', 'localhost:37017']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0723 22:16:31.413582 281473759148736 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0723 22:16:31.414247 281473759148736 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40479', 'localhost:40225', 'localhost:35819', 'localhost:37017']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0723 22:16:31.414485 281473759148736 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:40479', 'localhost:40225', 'localhost:35819', 'localhost:37017']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0723 22:16:31.604533 281473759148736 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: I0723 22:16:31.599009 281473759148736 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0723 22:16:31.608391 281473759148736 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I0723 22:16:31.609632 281473759148736 failure_handling.py:674] Start watcher for local signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0723 22:16:31.609919 281473759148736 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0723 22:16:31.610284 281473759148736 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 36 [worker-3]: I0723 22:16:31.610499 281473759148736 failure_handler_test.py:197] Start training at 36 [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-1]: I0723 22:16:31.622655 281473759148736 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:training restarted [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I0723 22:16:31.617898 281473759148736 failure_handler_test.py:207] training restarted [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-0]: I0723 22:16:31.636637 281473759148736 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0723 22:16:31.636991 281473759148736 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: I0723 22:16:31.621682 281473759148736 failure_handling.py:674] Start watcher for local signal. [worker-1]: I0723 22:16:31.623396 281473759148736 failure_handling.py:674] Start watcher for local signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0723 22:16:31.623649 281473759148736 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: I0723 22:16:31.622052 281473759148736 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Instructions for updating: [worker-1]: W0723 22:16:31.624006 281473759148736 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0723 22:16:31.637341 281473759148736 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: INFO:tensorflow:Start training at 36 [worker-0]: Instructions for updating: [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: I0723 22:16:31.624215 281473759148736 failure_handler_test.py:197] Start training at 36 [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: Instructions for updating: [worker-1]: INFO:tensorflow:training restarted [worker-0]: INFO:tensorflow:Start training at 36 [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0723 22:16:31.622428 281473759148736 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 36 [worker-2]: I0723 22:16:31.622639 281473759148736 failure_handler_test.py:197] Start training at 36 [worker-2]: INFO:tensorflow:training restarted [worker-2]: I0723 22:16:31.625904 281473759148736 failure_handler_test.py:207] training restarted [worker-1]: I0723 22:16:31.626732 281473759148736 failure_handler_test.py:207] training restarted [worker-0]: I0723 22:16:31.637551 281473759148736 failure_handler_test.py:197] Start training at 36 [worker-0]: INFO:tensorflow:training restarted [worker-0]: I0723 22:16:31.641512 281473759148736 failure_handler_test.py:207] training restarted [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:31.829421 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:31.853342 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:31.852402 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:31.864122 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:31.978075 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:31.978526 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:31.989421 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:31.997303 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:32.071036 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:32.077570 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:32.099608 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:32.105795 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:32.173363 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:32.174039 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:32.174186 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:32.174515 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:32.254398 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:32.256936 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:32.262107 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:32.254189 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffb33092d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0723 22:16:32.313218 281473759148736 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffb33092d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffb33092d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0723 22:16:32.315188 281473759148736 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffb33092d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffb330d2d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: I0723 22:16:32.327633 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: W0723 22:16:32.328059 281473759148736 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffb330d2d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: I0723 22:16:32.327809 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffb33092d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0723 22:16:32.336663 281473759148736 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffb33092d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:32.350490 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:32.357315 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffb330a200> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0723 22:16:32.426884 281473759148736 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffb330a200> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffb330a200> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0723 22:16:32.433255 281473759148736 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffb330a200> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffb330e200> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0723 22:16:32.440994 281473759148736 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffb330e200> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffb330a200> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0723 22:16:32.448107 281473759148736 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffb330a200> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:32.460498 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:32.482167 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:32.566419 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:32.601620 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:32.849442 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:32.851181 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:32.864448 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:32.867498 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:33.074593 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:33.076040 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:33.084337 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:33.114415 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-3]: INFO:tensorflow:epoch 2 finished [worker-0]: I0723 22:16:33.166446 281473759148736 failure_handler_test.py:195] epoch 2 finished [worker-3]: I0723 22:16:33.166864 281473759148736 failure_handler_test.py:195] epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0723 22:16:33.168188 281473759148736 failure_handler_test.py:195] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0723 22:16:33.168319 281473759148736 failure_handler_test.py:195] epoch 2 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:33.177950 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:33.179151 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:33.185596 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:33.185780 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:33.274791 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:33.276222 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:33.286863 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:33.299166 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:33.391942 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:33.401920 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:33.402447 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:33.401895 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:33.502254 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:33.511129 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:33.518251 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:33.505739 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:33.602505 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:33.617893 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:33.609644 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:33.609841 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:33.744450 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:33.766843 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:33.776442 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:33.944258 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:34.029191 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:34.032422 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:34.051325 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:34.079340 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:34.151388 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:34.162112 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:34.173093 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:34.252562 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:34.339490 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:34.371667 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:34.361984 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:34.360314 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:34.496763 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:34.497731 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:34.527669 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:34.531963 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:34.632168 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:34.641795 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:34.661597 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:34.682416 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:34.891238 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:34.891057 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:34.914366 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:34.926596 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:35.052987 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:35.061686 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:35.073025 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:35.192870 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:35.271666 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:35.285576 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:35.301792 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:35.292264 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:35.472783 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:35.461508 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:35.492399 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:35.597319 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-3]: I0723 22:16:35.677406 281473759148736 failure_handler_test.py:195] epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-0]: I0723 22:16:35.677574 281473759148736 failure_handler_test.py:195] epoch 3 finished [worker-1]: I0723 22:16:35.678716 281473759148736 failure_handler_test.py:195] epoch 3 finished [worker-2]: I0723 22:16:35.679920 281473759148736 failure_handler_test.py:195] epoch 3 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:35.702697 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:35.712005 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:35.742070 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:35.722027 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:35.941879 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:35.932461 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:35.929362 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:35.973038 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:36.219545 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:36.238710 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:36.261871 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:36.262155 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:36.379048 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:36.392570 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:36.402181 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:36.401896 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:36.521969 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:36.541930 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:36.582171 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:36.597367 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:36.712664 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:36.707746 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:36.722369 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:36.708179 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:37.061578 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:37.082746 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:37.072754 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:37.087549 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:37.255274 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:37.247071 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:37.261569 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:37.271545 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:37.430791 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:37.439695 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:37.439984 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:37.453929 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:37.557253 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:37.565180 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:37.572542 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:37.622050 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:37.732704 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:37.742208 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:37.747647 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:37.761756 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:38.009963 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:38.014741 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:38.042775 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:38.047397 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:38.149529 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:38.162403 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:38.181462 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:38.231938 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:38.299266 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:38.332778 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:38.333715 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:38.347830 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:38.461924 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:38.472386 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:38.474264 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:38.474411 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 4 finished [worker-0]: I0723 22:16:38.649924 281473759148736 failure_handler_test.py:195] epoch 4 finished [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I0723 22:16:38.655211 281473759148736 failure_handler_test.py:195] epoch 4 finished [worker-3]: INFO:tensorflow:epoch 4 finished [worker-3]: I0723 22:16:38.656611 281473759148736 failure_handler_test.py:195] epoch 4 finished [worker-1]: INFO:tensorflow:epoch 4 finished [worker-1]: I0723 22:16:38.668546 281473759148736 failure_handler_test.py:195] epoch 4 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:38.669761 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:38.673582 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:38.680564 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:38.695657 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:38.758381 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:38.764033 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:38.772098 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:38.787858 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:38.852089 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:38.854154 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:38.877492 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:38.864104 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:38.963515 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:38.963678 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:38.981497 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:38.998167 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:39.084800 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:39.092607 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:39.092682 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:39.081645 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:39.170183 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:39.189287 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:39.178947 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:39.198659 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:39.268150 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:39.273206 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:39.268872 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:39.274021 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:39.360502 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:39.360925 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:39.361898 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:39.362922 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:39.424367 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:39.425421 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:39.428426 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:39.424785 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:39.500175 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:39.507508 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:39.516572 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:39.522145 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:39.586512 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:39.587343 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:39.597143 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:39.611226 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:39.669949 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:39.670395 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:39.671095 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:39.671465 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:39.802388 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:39.825097 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:39.827488 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:39.842152 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:39.920320 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:39.917385 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:39.928342 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:39.935771 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:39.995862 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:39.997113 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:40.011498 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:40.032692 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 5 finished [worker-0]: INFO:tensorflow:epoch 5 finished [worker-3]: I0723 22:16:40.090232 281473759148736 failure_handler_test.py:195] epoch 5 finished [worker-0]: I0723 22:16:40.090359 281473759148736 failure_handler_test.py:195] epoch 5 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:40.101612 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:40.102438 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 5 finished [worker-1]: INFO:tensorflow:epoch 5 finished [worker-1]: I0723 22:16:40.107074 281473759148736 failure_handler_test.py:195] epoch 5 finished [worker-2]: I0723 22:16:40.107084 281473759148736 failure_handler_test.py:195] epoch 5 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:40.118508 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:40.118516 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:40.194237 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:40.194768 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:40.197535 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:40.198783 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:40.293394 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:40.293489 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:40.297822 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:40.302385 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:40.369585 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:40.375351 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:40.373741 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:40.369589 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:40.436423 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:40.441885 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:40.442826 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:40.452308 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:40.538148 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:40.562656 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:40.551127 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:40.572073 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:40.649900 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:40.663855 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:40.666827 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:40.670368 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:40.730037 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:40.731061 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:40.736088 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:40.732636 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:40.796308 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:40.796859 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:40.808758 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:40.810557 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:40.903267 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:40.918969 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:40.917560 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:40.951702 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:41.309702 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:41.303025 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:41.310500 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:41.328670 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:41.433706 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:41.424839 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:41.465241 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:41.461655 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:41.589148 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:41.597409 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:41.605636 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:41.651996 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:41.714729 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:41.730536 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:41.718160 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:41.771684 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:41.943462 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:41.964845 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:41.963685 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:41.998919 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 6 finished [worker-0]: INFO:tensorflow:epoch 6 finished [worker-2]: INFO:tensorflow:epoch 6 finished [worker-3]: I0723 22:16:42.095572 281473759148736 failure_handler_test.py:195] epoch 6 finished [worker-0]: I0723 22:16:42.095690 281473759148736 failure_handler_test.py:195] epoch 6 finished [worker-2]: I0723 22:16:42.095988 281473759148736 failure_handler_test.py:195] epoch 6 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:42.107447 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:42.107239 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 6 finished [worker-1]: I0723 22:16:42.116155 281473759148736 failure_handler_test.py:195] epoch 6 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:42.106673 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:42.171872 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:42.305463 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:42.321635 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:42.351031 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:42.351890 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:42.447353 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:42.448450 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:42.449239 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:42.447291 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:42.547992 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:42.556635 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:42.557260 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:42.570194 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:42.652965 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:42.685594 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:42.700647 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:42.722167 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:42.791758 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:42.792544 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:42.809276 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:42.821073 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:42.909889 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:42.912429 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:42.917227 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:43.025463 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:43.151926 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:43.151898 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:43.171830 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:43.177044 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:43.287787 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:43.325836 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:43.321429 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:43.349286 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:43.441204 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:43.452893 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:43.459003 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:43.471429 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:43.539006 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:43.539102 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:43.539502 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:43.544511 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:43.605629 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:43.610826 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:43.606576 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:43.640886 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:43.793215 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:43.795873 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:43.799246 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:43.799124 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:43.883042 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:43.889266 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:43.892428 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:43.905202 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:43.968003 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:43.980145 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:43.987353 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:43.991908 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 7 finished [worker-3]: I0723 22:16:44.145363 281473759148736 failure_handler_test.py:195] epoch 7 finished [worker-2]: INFO:tensorflow:epoch 7 finished [worker-2]: I0723 22:16:44.148382 281473759148736 failure_handler_test.py:195] epoch 7 finished [worker-2]: INFO:tensorflow:Training finished. [worker-0]: INFO:tensorflow:epoch 7 finished [worker-2]: I0723 22:16:44.149802 281473759148736 failure_handler_test.py:245] Training finished. [worker-1]: INFO:tensorflow:epoch 7 finished [worker-1]: I0723 22:16:44.155679 281473759148736 failure_handler_test.py:195] epoch 7 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I0723 22:16:44.157142 281473759148736 failure_handler_test.py:245] Training finished. [worker-0]: I0723 22:16:44.147878 281473759148736 failure_handler_test.py:195] epoch 7 finished [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I0723 22:16:44.150544 281473759148736 failure_handler_test.py:245] Training finished. [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I0723 22:16:44.176359 281473759148736 failure_handler_test.py:245] Training finished. I0723 22:16:45.147851 281473640004288 multi_process_runner.py:646] worker-0 exit code: 0 I0723 22:16:45.148175 281473640004288 multi_process_runner.py:646] worker-1 exit code: 0 I0723 22:16:45.148351 281473640004288 multi_process_runner.py:646] worker-2 exit code: 0 I0723 22:16:45.148513 281473640004288 multi_process_runner.py:646] worker-3 exit code: 0 I0723 22:16:45.150964 281473640004288 multi_process_runner.py:662] Joining log reading threads. I0723 22:16:45.151273 281473640004288 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker): 20.76s I0723 22:16:45.452601 281473640004288 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker): 20.76s [ OK ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_False_inputarg_manager_strategyoption_MWMSmultiworker [ RUN ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 33341 I0723 22:16:45.454752 281473640004288 test_util.py:3813] Using local port 33341 INFO:tensorflow:Using local port 39161 I0723 22:16:45.455288 281473640004288 test_util.py:3813] Using local port 39161 INFO:tensorflow:Using local port 39121 I0723 22:16:45.455708 281473640004288 test_util.py:3813] Using local port 39121 INFO:tensorflow:Using local port 42845 I0723 22:16:45.456133 281473640004288 test_util.py:3813] Using local port 42845 INFO:tensorflow:Cluster starting. I0723 22:16:45.507462 281473640004288 failure_handler_test.py:297] Cluster starting. [worker-2]: I0723 22:16:45.644688 281473759148736 multi_process_runner.py:840] Subprocess with PID 2341984 (worker, 2) is now being started. [worker-1]: I0723 22:16:45.646975 281473759148736 multi_process_runner.py:840] Subprocess with PID 2341950 (worker, 1) is now being started. [worker-1]: I0723 22:16:45.647528 281473759148736 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:33341", "localhost:39161", "localhost:39121", "localhost:42845"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: I0723 22:16:45.651482 281473759148736 multi_process_runner.py:840] Subprocess with PID 2341814 (worker, 0) is now being started. [worker-0]: I0723 22:16:45.652010 281473759148736 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:33341", "localhost:39161", "localhost:39121", "localhost:42845"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-2]: I0723 22:16:45.645257 281473759148736 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:33341", "localhost:39161", "localhost:39121", "localhost:42845"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0723 22:16:45.675807 281473759148736 multi_process_runner.py:840] Subprocess with PID 2341999 (worker, 3) is now being started. [worker-0]: 2023-07-23 22:16:45.689433: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:33341 [worker-3]: I0723 22:16:45.676417 281473759148736 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:33341", "localhost:39161", "localhost:39121", "localhost:42845"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: 2023-07-23 22:16:45.700136: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 13635072706712752348 [worker-2]: 2023-07-23 22:16:45.725076: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:39121 [worker-1]: 2023-07-23 22:16:45.723738: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:39161 [worker-2]: 2023-07-23 22:16:45.750760: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0723 22:16:45.752994 281473759148736 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0723 22:16:45.806655 281473759148736 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0723 22:16:45.807233 281473759148736 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33341', 'localhost:39161', 'localhost:39121', 'localhost:42845']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0723 22:16:45.807459 281473759148736 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33341', 'localhost:39161', 'localhost:39121', 'localhost:42845']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: 2023-07-23 22:16:45.728672: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: 2023-07-23 22:16:45.700357: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-0]: 2023-07-23 22:16:45.718267: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 17105666892810731874 [worker-1]: I0723 22:16:45.752994 281473759148736 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: 2023-07-23 22:16:45.728434: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 11177965927764000441 [worker-3]: 2023-07-23 22:16:45.713238: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:42845 [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-0]: 2023-07-23 22:16:45.750506: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:551] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 14108154931902616267 [worker-3]: 2023-07-23 22:16:45.718482: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:299] Coordination agent has successfully connected. [worker-1]: I0723 22:16:45.806572 281473759148736 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Check health not enabled. [worker-0]: I0723 22:16:45.752929 281473759148736 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I0723 22:16:45.753281 281473759148736 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: I0723 22:16:45.807161 281473759148736 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33341', 'localhost:39161', 'localhost:39121', 'localhost:42845']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-0]: I0723 22:16:45.806240 281473759148736 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-1]: I0723 22:16:45.807389 281473759148736 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33341', 'localhost:39161', 'localhost:39121', 'localhost:42845']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Check health not enabled. [worker-3]: I0723 22:16:45.806304 281473759148736 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-0]: I0723 22:16:45.806911 281473759148736 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33341', 'localhost:39161', 'localhost:39121', 'localhost:42845']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0723 22:16:45.806948 281473759148736 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: I0723 22:16:45.807144 281473759148736 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33341', 'localhost:39161', 'localhost:39121', 'localhost:42845']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33341', 'localhost:39161', 'localhost:39121', 'localhost:42845']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0723 22:16:45.807179 281473759148736 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:33341', 'localhost:39161', 'localhost:39121', 'localhost:42845']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0723 22:16:49.050639 281473759148736 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for local signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0723 22:16:49.050328 281473759148736 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: I0723 22:16:49.051583 281473759148736 failure_handling.py:674] Start watcher for local signal. [worker-2]: I0723 22:16:49.050501 281473759148736 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: I0723 22:16:49.050497 281473759148736 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: INFO:tensorflow:Start watcher for local signal. [worker-2]: INFO:tensorflow:Start watcher for local signal. [worker-1]: INFO:tensorflow:Start watcher for local signal. [worker-3]: I0723 22:16:49.051858 281473759148736 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: I0723 22:16:49.051716 281473759148736 failure_handling.py:674] Start watcher for local signal. [worker-2]: I0723 22:16:49.051280 281473759148736 failure_handling.py:674] Start watcher for local signal. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: I0723 22:16:49.052001 281473759148736 failure_handling.py:674] Start watcher for local signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: Instructions for updating: [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0723 22:16:49.052126 281473759148736 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: I0723 22:16:49.052320 281473759148736 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: I0723 22:16:49.051532 281473759148736 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: W0723 22:16:49.052236 281473759148736 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Instructions for updating: [worker-1]: Instructions for updating: [worker-2]: W0723 22:16:49.051905 281473759148736 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-0]: W0723 22:16:49.052538 281473759148736 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: I0723 22:16:49.052449 281473759148736 failure_handler_test.py:197] Start training at 0 [worker-1]: W0723 22:16:49.052714 281473759148736 deprecation.py:50] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py:198: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Instructions for updating: [worker-0]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0723 22:16:49.052923 281473759148736 failure_handler_test.py:197] Start training at 0 [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0723 22:16:49.052754 281473759148736 failure_handler_test.py:197] Start training at 0 [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0723 22:16:49.052114 281473759148736 failure_handler_test.py:197] Start training at 0 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:49.255598 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:49.326955 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:49.342187 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:49.360272 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:49.440761 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:49.467255 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:49.468226 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:sending sigterm [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I0723 22:16:50.598089 281473640004288 failure_handler_test.py:302] sending sigterm INFO:tensorflow:time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker): 14.66s I0723 22:17:00.114759 281473640004288 test_util.py:2478] time(__main__.PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker): 14.66s [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:49.467354 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:49.542305 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:49.542998 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:49.542451 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [ FAILED ] PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker [worker-1]: I0723 22:16:49.608923 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:49.609164 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:49.608209 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 ====================================================================== ERROR: test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker (__main__.PreemptionCheckpointTest) PreemptionCheckpointTest.test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker test_preemption_checkpointing_test_apiwrappingtrain_True_inputarg_manager_strategyoption_MWMSmultiworker(api_wrapping_train=True, input_arg='manager', strategy_option='MWMS_multi_worker') ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handler_test.py", line 304, in test_preemption_checkpointing os.kill(mpr.get_process_id('worker', killed_worker), signal.SIGTERM) ProcessLookupError: [Errno 3] No such process ---------------------------------------------------------------------- Ran 4 tests in 43.280s FAILED (errors=1) [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:49.541989 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:49.674474 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:49.672385 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:49.672396 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffb3301240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffb3301240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: I0723 22:16:49.609235 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffb3301240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0723 22:16:49.724800 281473759148736 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffb3301240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0723 22:16:49.724714 281473759148736 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffb3301240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: W0723 22:16:49.724364 281473759148736 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffb3301240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:49.672661 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:49.735195 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:49.736378 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:49.735289 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffffb3304790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffb3302170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffb3302170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffb3302170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0723 22:16:49.724558 281473759148736 polymorphic_function.py:156] 5 out of the last 5 calls to .wrapped_fn at 0xffffb3304790> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0723 22:16:49.782858 281473759148736 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffb3302170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0723 22:16:49.782817 281473759148736 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffb3302170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: W0723 22:16:49.782599 281473759148736 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffb3302170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:49.735408 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:49.794290 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:49.793591 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:49.793739 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffffb3306170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:49.852692 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: W0723 22:16:49.782796 281473759148736 polymorphic_function.py:156] 6 out of the last 6 calls to .wrapped_fn at 0xffffb3306170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: I0723 22:16:49.853468 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:49.851291 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:50.008093 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:49.794947 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:49.852580 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:50.021037 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:50.097685 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:50.035937 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:50.097594 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:50.264815 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:50.264756 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:49.912431 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:50.321718 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:50.098220 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:50.323217 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:50.204564 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:50.378966 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:50.378861 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:50.264639 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:50.265218 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:50.321722 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:50.435867 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:50.321892 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:50.378932 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:50.494596 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:50.378668 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:50.436821 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 0 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:50.542270 281473759148736 failure_handler_test.py:195] epoch 0 finished [worker-1]: I0723 22:16:50.435566 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:50.435563 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:50.552886 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:50.494948 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:50.494461 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 0 finished [worker-2]: INFO:tensorflow:epoch 0 finished [worker-0]: I0723 22:16:50.610552 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:50.542472 281473759148736 failure_handler_test.py:195] epoch 0 finished [worker-2]: I0723 22:16:50.542422 281473759148736 failure_handler_test.py:195] epoch 0 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:50.667525 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:50.554890 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:50.552756 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:50.726357 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:50.610743 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:50.610517 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:50.783622 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:50.668034 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:50.667415 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:50.726756 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:50.726247 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:50.842217 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:50.784468 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:50.783497 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:50.899581 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:50.841998 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:50.957427 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:50.899049 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:51.012568 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:50.494556 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:50.956181 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:51.067827 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 0 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:50.542109 281473759148736 failure_handler_test.py:195] epoch 0 finished [worker-2]: I0723 22:16:50.842127 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:51.125504 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:51.011975 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:50.899463 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:50.552801 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:51.181523 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:50.956629 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:51.069164 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:51.237698 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:50.610556 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:51.294161 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:51.124556 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:50.667481 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:51.012465 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:51.350480 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:50.726787 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:51.181346 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 1 finished [worker-2]: I0723 22:16:51.067775 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:51.396168 281473759148736 failure_handler_test.py:195] epoch 1 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:50.783512 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:51.125378 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:51.406997 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:51.181456 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:50.840619 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:51.237734 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:50.901133 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:51.463266 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:51.294126 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:50.956702 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:51.236719 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:51.521011 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:51.350354 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 1 finished [worker-3]: I0723 22:16:51.012525 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:51.577201 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:51.396295 281473759148736 failure_handler_test.py:195] epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:51.635100 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:51.406954 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:51.293654 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:51.463221 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:51.692804 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:51.349989 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:51.520821 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:51.396204 281473759148736 failure_handler_test.py:195] epoch 1 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:51.577675 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:51.407168 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:51.748462 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:51.462533 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:51.804842 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:51.520706 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:51.635078 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:51.860486 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:51.917355 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:51.691537 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:51.577197 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:51.634203 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:51.067831 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:51.691571 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:51.125410 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:51.181463 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:51.238136 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:51.294144 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:51.973786 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:51.350423 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:52.030497 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:51.395980 281473759148736 failure_handler_test.py:195] epoch 1 finished [worker-0]: I0723 22:16:52.087280 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:51.407002 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:52.144415 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:51.463250 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:52.202247 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-3]: I0723 22:16:51.519302 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:52.246735 281473759148736 failure_handler_test.py:195] epoch 2 finished [worker-2]: I0723 22:16:51.748437 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:52.257193 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:51.579347 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:52.313314 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:52.366392 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:52.420618 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:52.475127 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:52.528944 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:52.583573 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:52.636974 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:52.690767 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:52.743132 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:52.795835 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:52.849428 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:52.904136 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:52.957941 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:53.010607 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 3 finished [worker-0]: I0723 22:16:53.054058 281473759148736 failure_handler_test.py:195] epoch 3 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:53.063095 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:53.115849 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:53.168794 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:53.223556 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:53.277703 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:53.330498 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:53.381999 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:53.436962 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:53.491155 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:53.543746 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:53.597547 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:53.650604 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:53.703058 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:53.756960 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:53.808431 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 4 finished [worker-0]: I0723 22:16:53.851544 281473759148736 failure_handler_test.py:195] epoch 4 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:53.860809 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:51.804777 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:51.635120 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:51.691607 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:51.747389 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:53.913857 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:51.748907 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:53.966338 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:51.860469 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:51.803995 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:51.804821 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:51.860735 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:51.916866 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:51.918485 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:51.973926 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:51.860442 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:51.973703 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:52.029798 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:54.018803 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:52.030430 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:51.917309 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:52.086987 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:52.087028 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:54.071756 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:51.973751 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:52.142516 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:52.030466 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:54.124403 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:52.143163 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:52.201052 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:54.176689 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0723 22:16:52.246831 281473759148736 failure_handler_test.py:195] epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:52.256690 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:54.230234 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:52.312230 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:52.365997 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:54.282982 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:52.419571 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:54.335012 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:52.474313 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:54.387176 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:52.528156 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:54.439285 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:52.582577 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:54.492139 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:52.636078 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:54.545303 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:52.690251 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:52.201372 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:52.743066 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:54.597806 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:52.087006 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 5 finished [worker-2]: I0723 22:16:52.246925 281473759148736 failure_handler_test.py:195] epoch 2 finished [worker-1]: I0723 22:16:52.795877 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:52.143203 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:54.641192 281473759148736 failure_handler_test.py:195] epoch 5 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:52.849077 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:52.257225 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:54.650692 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:52.199885 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:52.904138 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:54.703776 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:52.313282 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:52.246584 281473759148736 failure_handler_test.py:195] epoch 2 finished [worker-1]: I0723 22:16:52.958026 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.010698 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:54.757413 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 3 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:52.366895 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.054166 281473759148736 failure_handler_test.py:195] epoch 3 finished [worker-0]: I0723 22:16:54.809560 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.063191 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:54.863356 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:52.258862 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:52.420577 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:52.313296 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.116068 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:54.914767 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.168794 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:52.366856 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:52.475094 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.223594 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:52.528906 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.277801 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:54.967470 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:52.583557 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:52.420609 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:55.020586 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.330541 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:52.636578 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:55.073040 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:52.475097 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.382162 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:52.690723 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:55.125344 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:52.528920 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.437001 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:52.743588 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:55.177123 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:52.583581 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:52.795895 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:52.636617 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:52.690593 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:52.849727 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:55.229377 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:52.743558 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:55.281981 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:52.795612 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:55.335624 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:55.389394 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:52.848212 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:52.903710 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 6 finished [worker-3]: I0723 22:16:52.905256 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:55.433292 281473759148736 failure_handler_test.py:195] epoch 6 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:52.958475 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:52.958381 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:55.442487 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:55.496379 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:53.011238 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 3 finished [worker-0]: I0723 22:16:55.549845 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:53.011068 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:53.054246 281473759148736 failure_handler_test.py:195] epoch 3 finished [worker-3]: INFO:tensorflow:epoch 3 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:53.053909 281473759148736 failure_handler_test.py:195] epoch 3 finished [worker-0]: I0723 22:16:55.602879 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.491131 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:53.063395 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.543816 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:53.116055 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:55.656455 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:53.062966 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:55.710291 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.597627 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:53.168396 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.650622 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:55.763228 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:53.223452 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:55.816877 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:53.278552 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.703139 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:55.869941 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.756961 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:53.330831 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:55.924216 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.808442 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 4 finished [worker-0]: I0723 22:16:55.976621 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:53.382626 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.851640 281473759148736 failure_handler_test.py:195] epoch 4 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:56.031336 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:53.437826 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.860808 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:56.083226 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:53.491868 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.913918 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:53.544275 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:53.116041 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:53.598133 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:53.651180 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:53.703704 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:53.757471 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:53.808662 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I0723 22:16:53.851701 281473759148736 failure_handler_test.py:195] epoch 4 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:53.861186 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:53.914387 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:53.966786 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:54.019510 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:54.072096 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:54.125271 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:56.137925 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:53.966451 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:54.177227 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:53.168276 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:56.192916 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:54.018841 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:54.231279 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:53.223474 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 7 finished [worker-0]: I0723 22:16:56.239809 281473759148736 failure_handler_test.py:195] epoch 7 finished [worker-3]: I0723 22:16:53.278556 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:54.283603 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Training finished. [worker-1]: I0723 22:16:54.071817 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0723 22:16:56.241575 281473759148736 failure_handler_test.py:245] Training finished. [worker-3]: I0723 22:16:53.330621 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:54.335754 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:54.124436 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:53.382627 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:54.387634 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:54.176703 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:53.437789 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:54.440497 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:54.230337 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:53.490256 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:54.492697 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:54.283017 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:53.545852 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:54.546144 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:54.335113 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:53.598603 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:54.598237 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:54.387209 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 5 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:53.651151 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:54.641396 281473759148736 failure_handler_test.py:195] epoch 5 finished [worker-1]: I0723 22:16:54.439379 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:54.651209 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:53.704045 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:54.492186 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:54.704257 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:53.757411 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:54.545419 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:54.758052 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:53.808671 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:54.597898 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-2]: I0723 22:16:54.810127 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 5 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:53.851395 281473759148736 failure_handler_test.py:195] epoch 4 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:54.641309 281473759148736 failure_handler_test.py:195] epoch 5 finished [worker-3]: I0723 22:16:53.860891 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:54.863700 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:53.914379 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:54.650762 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:54.915093 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:53.966655 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:54.703804 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:54.968244 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:54.019416 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:54.757620 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:55.020989 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:54.071851 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:54.809678 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:55.073704 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:54.123183 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:54.863492 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:55.125621 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:54.178723 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:54.914793 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:55.177655 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:54.230750 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:54.967583 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:55.230027 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:54.283544 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:55.020653 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:55.282873 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:54.335207 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:55.073146 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:55.336076 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:54.387498 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:55.125380 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:55.389875 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 6 finished [worker-3]: I0723 22:16:54.440083 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:55.433464 281473759148736 failure_handler_test.py:195] epoch 6 finished [worker-1]: I0723 22:16:55.177257 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:54.492699 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:55.442945 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:55.229409 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:55.282006 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:55.335708 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:54.546168 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:55.497020 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:55.389608 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 6 finished [worker-3]: I0723 22:16:54.598068 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:55.433408 281473759148736 failure_handler_test.py:195] epoch 6 finished [worker-2]: I0723 22:16:55.550161 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 5 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:54.641057 281473759148736 failure_handler_test.py:195] epoch 5 finished [worker-1]: I0723 22:16:55.442594 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:55.603523 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:54.651190 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:55.496445 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:55.657396 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:55.549903 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:55.710543 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:55.602997 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:54.704182 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:55.764103 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:55.656492 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:54.756515 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:55.817526 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:55.710322 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:54.811673 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:55.870702 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:55.763345 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:54.863507 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:55.924860 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:55.816996 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:54.915032 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:55.977345 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:55.869998 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:54.968276 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:56.031868 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:55.924314 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:55.020952 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:56.084146 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:55.976760 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:55.073679 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:56.138221 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:56.031529 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:55.125493 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:56.194327 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:56.083340 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 7 finished [worker-3]: I0723 22:16:55.177495 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:56.240124 281473759148736 failure_handler_test.py:195] epoch 7 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:56.138271 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Training finished. [worker-3]: I0723 22:16:55.230037 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0723 22:16:56.241933 281473759148736 failure_handler_test.py:245] Training finished. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:56.192941 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:55.282892 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 7 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:56.239924 281473759148736 failure_handler_test.py:195] epoch 7 finished [worker-3]: I0723 22:16:55.336062 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Training finished. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0723 22:16:56.241338 281473759148736 failure_handler_test.py:245] Training finished. [worker-3]: I0723 22:16:55.388330 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 6 finished [worker-3]: I0723 22:16:55.433163 281473759148736 failure_handler_test.py:195] epoch 6 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:55.444155 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:55.497027 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:55.550195 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:55.603490 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:55.657411 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:55.710491 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:55.764108 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:55.817514 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:55.870726 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:55.924864 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:55.977365 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:56.030444 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:56.086016 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:56.138130 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0723 22:16:56.194287 281473759148736 cross_device_ops.py:1154] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 7 finished [worker-3]: I0723 22:16:56.239854 281473759148736 failure_handler_test.py:195] epoch 7 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I0723 22:16:56.241806 281473759148736 failure_handler_test.py:245] Training finished. ================================================================================ //tensorflow/compiler/xla/python/pjrt_ifrt:xla_sharding_serdes_test FAILED TO BUILD //tensorflow/c:c_api_experimental_test PASSED in 30.0s //tensorflow/c:c_api_function_test PASSED in 29.4s //tensorflow/c:c_api_test_cpu PASSED in 37.9s //tensorflow/c:c_test PASSED in 33.2s //tensorflow/c:env_test_cpu PASSED in 28.7s //tensorflow/c:kernels_test_cpu PASSED in 60.5s //tensorflow/c:ops_test PASSED in 25.2s //tensorflow/c:tf_status_helper_test PASSED in 0.1s //tensorflow/c:while_loop_test PASSED in 36.0s //tensorflow/c/eager:c_api_cluster_test_cpu PASSED in 40.8s //tensorflow/c/eager:c_api_remote_function_test_cpu PASSED in 33.1s //tensorflow/c/eager:c_api_remote_test_cpu PASSED in 31.3s //tensorflow/c/eager:c_api_test_cpu PASSED in 37.3s //tensorflow/c/eager:custom_device_test PASSED in 28.6s //tensorflow/c/eager/parallel_device:parallel_device_lib_test PASSED in 25.3s //tensorflow/c/eager/parallel_device:parallel_device_remote_test PASSED in 32.4s //tensorflow/c/eager/parallel_device:parallel_device_test PASSED in 34.1s //tensorflow/c/experimental/filesystem/plugins/gcs:expiring_lru_cache_test PASSED in 0.3s //tensorflow/c/experimental/filesystem/plugins/gcs:ram_file_block_cache_test PASSED in 2.7s //tensorflow/c/experimental/grappler:grappler_test PASSED in 26.4s //tensorflow/c/experimental/next_pluggable_device:tensor_pjrt_buffer_util_test PASSED in 10.0s //tensorflow/c/experimental/ops/gen/common:case_format_test PASSED in 0.5s //tensorflow/c/experimental/ops/gen/cpp:cpp_generator_test PASSED in 0.7s //tensorflow/c/experimental/ops/gen/cpp/renderers:renderer_test PASSED in 0.5s //tensorflow/c/experimental/saved_model/core:constant_loading_test PASSED in 16.0s //tensorflow/c/experimental/saved_model/core:object_graph_traversal_test PASSED in 15.3s //tensorflow/c/experimental/saved_model/core:saved_variable_loading_test PASSED in 19.0s //tensorflow/c/experimental/saved_model/core:signature_flattening_test PASSED in 24.5s //tensorflow/c/experimental/saved_model/core:tf_concrete_function_loading_test PASSED in 11.7s //tensorflow/c/experimental/saved_model/core/ops:restore_ops_test PASSED in 14.9s //tensorflow/c/experimental/saved_model/core/ops:variable_ops_test PASSED in 16.9s //tensorflow/c/experimental/saved_model/internal:saved_model_api_test PASSED in 30.6s //tensorflow/c/experimental/stream_executor:stream_executor_test PASSED in 0.3s //tensorflow/c/kernels:bitcast_op_test PASSED in 0.9s //tensorflow/c/kernels:summary_op_benchmark_test PASSED in 1.0s //tensorflow/c/kernels:summary_op_test PASSED in 1.1s //tensorflow/c/kernels:tensor_shape_utils_test PASSED in 0.1s //tensorflow/cc:cc_op_gen_test PASSED in 0.8s //tensorflow/cc:client_client_session_test PASSED in 2.8s //tensorflow/cc:coordinator_test PASSED in 3.9s //tensorflow/cc:framework_cc_ops_test PASSED in 2.0s //tensorflow/cc:framework_gradient_checker_test PASSED in 1.7s //tensorflow/cc:framework_gradients_test PASSED in 4.8s //tensorflow/cc:framework_scope_test PASSED in 0.5s //tensorflow/cc:framework_while_gradients_test PASSED in 2.2s //tensorflow/cc:gradients_array_grad_test PASSED in 7.4s //tensorflow/cc:gradients_data_flow_grad_test PASSED in 1.5s //tensorflow/cc:gradients_functional_grad_test PASSED in 2.4s //tensorflow/cc:gradients_image_grad_test PASSED in 6.8s //tensorflow/cc:gradients_linalg_grad_test PASSED in 2.5s //tensorflow/cc:gradients_manip_grad_test PASSED in 1.7s //tensorflow/cc:gradients_math_grad_test PASSED in 5.3s //tensorflow/cc:gradients_nn_grad_test PASSED in 3.7s //tensorflow/cc:gradients_resource_variable_grad_test PASSED in 2.1s //tensorflow/cc:ops_const_op_test PASSED in 0.5s //tensorflow/cc:ops_while_loop_test PASSED in 3.0s //tensorflow/cc:queue_runner_test PASSED in 12.6s //tensorflow/cc/experimental/base/tests:tensor_test PASSED in 0.1s //tensorflow/cc/experimental/base/tests:tensorhandle_test PASSED in 32.2s //tensorflow/cc/experimental/libexport:load_test PASSED in 0.2s //tensorflow/cc/experimental/libexport:save_test PASSED in 0.1s //tensorflow/cc/experimental/libtf:libtf_module_test PASSED in 33.6s //tensorflow/cc/experimental/libtf:libtf_object_test PASSED in 0.7s //tensorflow/cc/experimental/libtf:libtf_perf_test PASSED in 0.2s //tensorflow/cc/experimental/libtf:libtf_runtime_test PASSED in 37.3s //tensorflow/cc/experimental/libtf:libtf_transform_test PASSED in 35.9s //tensorflow/cc/experimental/libtf:libtf_value_test PASSED in 0.6s //tensorflow/cc/experimental/libtf:libtf_visit_test PASSED in 0.4s //tensorflow/cc/experimental/libtf/impl:iostream_test PASSED in 0.1s //tensorflow/cc/experimental/libtf/impl:none_test PASSED in 16.3s //tensorflow/cc/experimental/libtf/impl:scalars_test PASSED in 0.2s //tensorflow/cc/experimental/libtf/impl:string_test PASSED in 0.9s //tensorflow/cc/experimental/libtf/impl:tensor_spec_test PASSED in 0.2s //tensorflow/cc/saved_model:bundle_v2_test PASSED in 0.1s //tensorflow/cc/saved_model:fingerprinting_test PASSED in 1.0s //tensorflow/cc/saved_model:metrics_test PASSED in 0.4s //tensorflow/cc/saved_model:reader_test PASSED in 0.1s //tensorflow/cc/saved_model:saved_model_bundle_lite_test PASSED in 6.0s //tensorflow/cc/saved_model:saved_model_bundle_test PASSED in 10.8s //tensorflow/cc/saved_model:util_test PASSED in 0.1s //tensorflow/cc/saved_model/experimental/tests:saved_model_api_test PASSED in 31.7s //tensorflow/cc/tools:freeze_saved_model_test PASSED in 1.8s //tensorflow/compiler/aot:codegen_test PASSED in 26.4s //tensorflow/compiler/jit:compilability_check_util_test PASSED in 21.2s //tensorflow/compiler/jit:deadness_analysis_test PASSED in 8.4s //tensorflow/compiler/jit:device_compilation_cache_test PASSED in 6.6s //tensorflow/compiler/jit:device_compilation_cluster_signature_test PASSED in 5.3s //tensorflow/compiler/jit:device_compilation_profiler_test PASSED in 23.2s //tensorflow/compiler/jit:device_compiler_client_test PASSED in 5.3s //tensorflow/compiler/jit:device_compiler_disable_test PASSED in 21.4s //tensorflow/compiler/jit:device_executable_persistor_test PASSED in 26.1s //tensorflow/compiler/jit:device_util_test PASSED in 5.9s //tensorflow/compiler/jit:encapsulate_util_test PASSED in 0.8s //tensorflow/compiler/jit:node_matchers_test PASSED in 1.1s //tensorflow/compiler/jit:resource_operation_safety_analysis_test PASSED in 8.4s //tensorflow/compiler/jit:shape_inference_test PASSED in 0.8s //tensorflow/compiler/jit:xla_activity_listener_test PASSED in 20.8s //tensorflow/compiler/jit:xla_cluster_util_test PASSED in 8.9s //tensorflow/compiler/jit:xla_compile_util_test PASSED in 6.9s //tensorflow/compiler/jit:xla_kernel_creator_test PASSED in 9.2s //tensorflow/compiler/jit:xla_launch_util_test PASSED in 22.0s //tensorflow/compiler/jit/tests:auto_clustering_test PASSED in 21.1s //tensorflow/compiler/mlir:mlir_graph_optimization_pass_test PASSED in 17.0s //tensorflow/compiler/mlir:register_common_dialects_test PASSED in 20.5s //tensorflow/compiler/mlir/lite:lstm_utils_test PASSED in 1.4s //tensorflow/compiler/mlir/lite:perception_ops_utils_test PASSED in 0.8s //tensorflow/compiler/mlir/lite:size_utils_test PASSED in 0.2s //tensorflow/compiler/mlir/lite:tftext_utils_test PASSED in 1.0s //tensorflow/compiler/mlir/lite/experimental/remat:rematerializer_test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac:execution_metadata_exporter_test PASSED in 5.4s //tensorflow/compiler/mlir/lite/experimental/tac/tests:compute-cost.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-gpu.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-nnapi.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/experimental/tac/tests:fold-constants-to-subgraph.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:get-alternative-subgraph.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:get-op-cost.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/experimental/tac/tests:pick-subgraphs.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/experimental/tac/tests:raise-target-subgraphs.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/lite/experimental/tac/tests:tac-filter.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/experimental/tac/tests:target-annotation.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:device-transform-nnapi.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:simple-graph.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/metrics:error_collector_inst_test PASSED in 1.0s //tensorflow/compiler/mlir/lite/quantization:numerical_utils_test PASSED in 0.2s //tensorflow/compiler/mlir/lite/quantization/lite:quantize_model_test PASSED in 10.7s //tensorflow/compiler/mlir/lite/quantization/lite:quantize_weights_test PASSED in 15.0s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_default.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_legacy.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant_4bit.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/quantization/tests:import_quant_stats.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/sparsity:sparsify_model_test PASSED in 1.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:compose-uniform-quantized-type.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:fold_broadcast.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:fuse_mhlo_convolution.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-inplaceupdate.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-skip-quantization-ops.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tf-fb-tf.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-add.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-broadcast_in_dim.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-clamp.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-compare.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-concat.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-constant.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-conv.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-dot.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-gather.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-max.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-mul.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-pad.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-reshape.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-rsqrt.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-scatter.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-sub.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-add.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-broadcast.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-clamp.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-concat.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-constant.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-conv.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-max.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-mul.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-pad.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-reshape.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-rsqrt.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-sub.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-allow-tf.mlir.test PASSED in 3.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-smuggle-resize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:optimize.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-clamp.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-concat.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-conv.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-division.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-logistic.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-multiply.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-reduce-window.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-resize-bilinear.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-subtract.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-tf-quantize.mlir.test PASSED in 18.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:unfuse_mhlo_batch_norm.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:uniform-quantized-stablehlo-to-tfl.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:analyze-variables.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:canonicalize.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:const-fold.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:decompose-hybrid-quantization.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:default_quant_params.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:dilated-conv.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:fuse-tftext.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:get-arithmetic-count.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests:guarantee_func_has_one_use.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:inlining.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:insert_call_once_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests:legalize-tensorlist.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:legalize-tf-assert.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:legalize-tf-hashtables.mlir.test PASSED in 2.9s //tensorflow/compiler/mlir/lite/tests:legalize-tf-no-runtime-verification.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests:legalize-tf-variables.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:legalize-tf-while.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:legalize-tf.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests:legalize_jax_random.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:lift_tflite_flex_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-default-to-single-batch.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-enable-dynamic-update-slice.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:modify_io_nodes.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:ops.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:optimize-after-quantization.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests:optimize.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/tests:optimize_functional_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:optimize_no_verify.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:optimize_op_order.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:partitioned-topological-sort.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests:pin-ops-with-side-effects.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:post-quantize-dynamic-range.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/lite/tests:post-quantize.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/lite/tests:prepare-composite-functions-tf.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-dynamic-range.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training-16bits.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-signed.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:prepare-quantize.mlir.test PASSED in 2.8s //tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant-4bit.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:prepare-tf-with-allowing-bf16-and-f16-type-legalization.mlir.test PASSED in 2.6s //tensorflow/compiler/mlir/lite/tests:prepare-tf.mlir.test PASSED in 2.9s //tensorflow/compiler/mlir/lite/tests:quantize-dynamic-range.mlir.test PASSED in 3.9s //tensorflow/compiler/mlir/lite/tests:quantize-numeric-verify.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:quantize-variables.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:quantize.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:raise-custom-ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:reduce_while_operands.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/tests:shape-inference.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:split-merged-operands.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests:tfl_while_op_licm.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:tfl_while_outline.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:trim-functions-tf.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:unfold-large-splat-constant.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.line.part.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.stack.part.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/end2end:add.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/end2end:back2back_fake_quant.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:control_flow_v1.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/end2end:conv_2d.pbtxt.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests/end2end:conv_2d_nchw.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:custom_opdef.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:disallow_stateful_partitioned_call.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel_4bit.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity_4bit.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:graph-input-node.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:graph_with_placeholder_with_default.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/end2end:if_op.pbtxt.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests/end2end:quant_stats.pbtxt.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul_disabled.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:basic_lstm.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:bucketize.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:constants.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:constants_offset.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:control_edges.mlir.test PASSED in 23.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:custom_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:custom_op_offset.mlir.test PASSED in 0.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:dynamic_shape.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:empty_input_output_names.json.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:external_constant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:if_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:import_json.json.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_arrays.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_output_names_attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:legacy_reshape.json.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.json.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:many_attribute_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:math.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:matmul.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:multi_output_op.json.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional_input.json.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:output_arrays.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning_function_input_as_output.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quant_stats.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quantization.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:reshape.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature_with_multiple_entry_points.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:simple.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:tf_variant_type.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_function_output.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_tensor.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:while_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2exec:tfl_while_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:basic_lstm.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:bucketize.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:custom_op_with_tflite_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:custom_tensorlist_reserve.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d_v2.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_builtin.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_custom.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex_enable_builtin.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:dynamic_shape_constant.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fake_quant.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_exclusively.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_complex128.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_f64.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_tflite_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected_v2.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:hashtable_resource.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:if_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:logical.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:low_bit_packing.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_asym_attr.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_quantized.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:math.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:metadata.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v2.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v3.mlir.test PASSED in 0.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:nn.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:numeric_verify.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:optional.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:quantization.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:reshape.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_output_override.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_multiple_entry_points.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_no_inputs.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_connected_control_nodes.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_unconnected_control_nodes.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf_v2.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tf_entry_function.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tfl_while_op.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:transpose_conv_optional.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:type_attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_lstm.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_rnn.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unranked_tensor.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unsorted_segment_prod.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_func.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:while_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/stablehlo:convert_tf_quant_to_mhlo_int_test PASSED in 8.9s //tensorflow/compiler/mlir/quantization/stablehlo:math_utils_test PASSED in 0.3s //tensorflow/compiler/mlir/quantization/stablehlo/tests:fill_quantization_options_test PASSED in 2.0s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:calibrator_singleton_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:custom_aggregator_op_test PASSED in 19.4s //tensorflow/compiler/mlir/quantization/tensorflow/cc:const_op_size_test PASSED in 1.8s //tensorflow/compiler/mlir/quantization/tensorflow/cc:constant_fold_test PASSED in 36.0s //tensorflow/compiler/mlir/quantization/tensorflow/cc:convert_asset_args_test PASSED in 6.2s //tensorflow/compiler/mlir/quantization/tensorflow/cc:save_variables_test PASSED in 0.4s //tensorflow/compiler/mlir/quantization/tensorflow/cc:status_macro_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/tensorflow/debugging:mlir_dump_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/tensorflow/python:concurrency_test PASSED in 51.7s //tensorflow/compiler/mlir/quantization/tensorflow/python:pywrap_quantize_model_test PASSED in 24.1s //tensorflow/compiler/mlir/quantization/tensorflow/python:representative_dataset_test PASSED in 9.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:cast_bf16_ops_to_f32.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_custom_aggregation_op_to_quant_stats.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_fake_quant_to_qdq.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_tpu_model_to_cpu.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:duplicate_shape_determining_constants.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_flow.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_xla.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_custom_aggregation_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_main_function.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_drq.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_weight_only.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_restore_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_save_op.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:issue_ids_of_custom_aggregation_ops.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_hashtable_ops_as_args.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq_min_elements.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_xla.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:mark_functions_noinline.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_duplicate_resource_ops.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_initializer_function_ops_to_main.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_save_function_ops_to_main.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:optimize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_lifting.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq_per_channel.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq_per_channel.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:preprocess_op.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:preprocess_op_weight_only.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_drq.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_weight_only.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_xla.mlir.test PASSED in 3.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_drq.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_xla.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:remove_var_init_by_const.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops_large_constants.mlir.test PASSED in 13.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:unfreeze_constants.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/utils:tf_to_xla_attribute_utils_test PASSED in 40.9s //tensorflow/compiler/mlir/stablehlo:stablehlo_test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow:bridge_logger_test PASSED in 6.1s //tensorflow/compiler/mlir/tensorflow:call_graph_util_test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow:cluster_util_test PASSED in 0.3s //tensorflow/compiler/mlir/tensorflow:convert_tensor_test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow:convert_type_test PASSED in 0.4s //tensorflow/compiler/mlir/tensorflow:device_util_test PASSED in 0.4s //tensorflow/compiler/mlir/tensorflow:dump_graph_test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow:dump_mlir_util_test PASSED in 18.4s //tensorflow/compiler/mlir/tensorflow:error_util_test PASSED in 0.3s //tensorflow/compiler/mlir/tensorflow:tf_mlir_translate_registration_test PASSED in 18.5s //tensorflow/compiler/mlir/tensorflow:tf_saved_model_test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow:tpu_rewrite_device_util_test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow:xla_rewrite_util_test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:add_functions_for_exported_names.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:annotate-parameter-replication.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:batchmatmul_to_einsum.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:breakup-islands.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:cannonicalize_ops_outside_compilation.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/tensorflow/tests:canonicalize.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:canonicalize_compile_and_replicate_attributes.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:check_control_dependencies.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:cluster_formation.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:cluster_ops_by_policy.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:cluster_outlining.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:cluster_tf_ops_pass.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:constant-fold.mlir.test PASSED in 2.8s //tensorflow/compiler/mlir/tensorflow/tests:constant_op_device_assignment.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:convert-tf-control-flow-to-scf.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:convert_control_to_data_outputs.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests:convert_launch_func_to_tf_call.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:convert_session_initializer_to_function.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:convert_to_legacy_compile_and_replicate_attributes.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:decompose_reduce_dataset.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:decompose_resource_ops.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:device_assignment.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:device_assignment_by_func_attr.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:device_attribute_to_launch.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:device_canonicalize.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:device_copy.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:drop_while_shape_invariant.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:einsum.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:embedding_pipelining.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:embedding_program_key.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:embedding_sequencing.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:empty-main.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:end-to-end-tpu-reshard-variables.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:executor_canonicalize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:executor_island_coarsening.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:executor_island_materialize_const.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:extract_head_tail_outside_compilation.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:extract_outside_compilation.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:extract_tpu_copy_with_dynamic_shape_op.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:fold-broadcast.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:freeze_variables.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:func-attr-invalid.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:func-attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-cfg.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-regions.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:functionalize-if-fail.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:functionalize-if.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:fused_kernel_matcher.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:gpu_fusion.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:graph_pruning.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:graph_pruning_preserve_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:group_by_dialect.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:guarantee-all-funcs-one-use.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:hoist_loop_invariant.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:hoist_replicate_invariant_resource_writes.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:host_launch_to_outside_compiled.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_invalid.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_saved_model.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tensorflow/tests:inlining.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:isolate-placer.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:launch_outlining.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute_legacy.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_60.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_70.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nchw.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nhwc.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_begin.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_end.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nchw.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nhwc.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:legalize_hlo.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_arg_control_dep.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_with_control_flow.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:localize_var_handles.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program_invalid.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:lower_quantized.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:lower_tf.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/tensorflow/tests:lower_variable_ops_to_ml_program.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:mark_input_output_aliases.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:mark_ops_for_outside_compilation.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:materialize_passthrough_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:merge_control_flow.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:mlprogram.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:name_anonymous_iterators.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:optimize-arg-operand-constraint.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:optimize.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:order_by_dialect.mlir.test PASSED in 11.3s //tensorflow/compiler/mlir/tensorflow/tests:outside_compiled_to_host_launch.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands_legacy.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:prepare_tpu_computation_for_tf_export.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args_functions.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:promote_var_handles_to_args.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:readonly_references_to_resources.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:region-control-flow-to-functional.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:remove_unused_arguments.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:remove_unused_while_results.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:replica_id_to_device_ordinal.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:replicate_invariant_op_hoisting.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:replicate_tensor_list_init_ops.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island_legacy.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:resource-alias-analysis-test.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:resource-device-inference.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:resource_analyzer.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:resource_inlining.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:resource_op_lifting.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tensorflow/tests:rewrite_tpu_embedding_ops.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:roundtrip-tf-executor.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:shape_inference.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:side-effect-analysis-test.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:sink_constant.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:split_into_island_per_op.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:stack_ops_decomposition.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:strip_noinline.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:strip_saved_module_metadata.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:strip_tf_attributes.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tensor_array_ops_decomposition.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tensor_list_ops_decomposition.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf-executor-to-functional.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tf-functional-to-executor.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf-ops.mlir.test PASSED in 4.5s //tensorflow/compiler/mlir/tensorflow/tests:tf-reduce-identity.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_map_and_batch.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_pmap_and_batch.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_index_selector.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops_invalid.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_invalid.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_location_roundtrip.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_printer.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_side_effect.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:tf_optimize.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_asset_sinking.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_deduplicate_bound_input_bindings.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_assets.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors_mutable_tensors.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init_fail.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables_invalid_session.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_mark_initialized_variables.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops_invalid.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors_interprocedural.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_remove_vars_in_session_initializer.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tf_side_effect.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_trait_folds.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tfrt_ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu-annotate-dynamic-shape-inputs.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tpu-cluster-cleanup-attributes.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu-dynamic-layout-pass.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tpu-merge-variables-with-execute.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tpu-multiple-while-body-func.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu-resource-read-for-write.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu-variable-runtime-reformatting.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_cluster_formation.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tpu_colocate_composite_resource_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_colocate_splits.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu_device_propagation.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:tpu_host_computation_expansion.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tpu_identity_pruning.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_parallel_execute_sink_resource_write.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tpu_partitioned_op_conversion.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_reorder_replicate_and_partitioned_inputs.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tpu_resource_partitioning.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu_rewrite.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tpu_sharding_identification.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_space_to_depth_pass.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_tail_with_tobool_op.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_update_embedding_enqueue_op_inputs.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu_validate_inputs.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:transpose-op.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:unroll-batch-matmul.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:update_control_dependencies.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:warn_when_using_deprecated_dumps.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:while_licm.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_deserialization.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_round_trip.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:xla_call_module_serialization.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:xla_cluster_formation.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:xla_inline_device_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:xla_rewrite.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:xla_rewrite_v2.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:xla_validate_iputs.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:add.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding-invalid.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding-hook.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:convert_mhlo_quant_to_int.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:mlir-module-serialized-str-attr.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:replicate-tensor-list-init-ops.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:result-sharding.mlir.test PASSED in 2.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr-invalid.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference-after-legalization.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:stablehlo_add.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:executor_tpuv1_island_coarsening.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:while_op.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:executor_tpuv1_inline_tpu_island.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:while_op.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:case_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:executor_tpuv1_outline_tpu_island.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:while_op.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:add.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-as-fetch.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-control-dep.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type-with-subtype.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-multi-data-type-with-subtype.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-retval-attrs.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:case_op.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:const-values.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:device-arg-retval-attr.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-input-shapes.pbtxt.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-value-attr.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-as-fetch.pbtxt.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-control-dep.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:force_shared_name_for_resource_ops.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:function-func-attr.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-if-ops.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-while-ops.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-control-ret.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-retval-of-arg.pbtxt.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-custom-operation.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-default-attr.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-device-retval.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-empty-tensor-content.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-func-attr.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-call.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-diff-island.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-same-island.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-defs.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-input-shapes.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-name-bug.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-resource-args.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-gradient-def.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-input-func-arg-name-collision.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-library.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-malformed.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-scalar-input.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-uint8-return.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-undefined-output.pbtxt.test PASSED in 2.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-version-info.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-while-loop.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:invalid-output-index.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:legacy-fed-input-without-inputs.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:merge_node_with_function.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:mlir_passthrough_op.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multi-output-feeds.pbtxt.test PASSED in 2.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multiple-use-next-iteration.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:node-locations.pbtxt.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes-attr.pbtxt.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes.pbtxt.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example_v2.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:partial-device-name.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:prune_unused_nodes.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:quint8-const.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:shape-attrs.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:stateful-attribute.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:string-attr.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:switch_n.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:target.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tensor-list.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tf-data-pipeline.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:unregistered_kernel.pbtxt.test PASSED in 8.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir/batch_use_same_function:saved_model.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graph:convert_tensor.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:aliasing_arg_attr.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:case.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:convert_tensor.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_shape_attr.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_size_attr.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:device-arg-retval-attr.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:export_main_to_flib.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:fetch_feed_names.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_attr.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_list_attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-control-ret.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-order.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args-handle-info.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-if-ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-while-ops.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:graph-as-function.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:infer_derived_attribute.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:invalid_input.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:legalized_name.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:missing-main.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:noop.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:optional_symbol_ref.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:output-shapes-attr.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example_v2.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:preserve-entry-func-names.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-type-attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-while-loop.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:shape_list_attr.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple_tf_dialect_op.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:stringescape.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:switchn.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-gradient-attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-legacy-call.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_add.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_identity_n.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_tpu_embedding_ops.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_list_attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_name.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_output_name.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:while-loop.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/tf_to_hlo_pipeline:sccp-post-shape-inference.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/tpu_bridge_v1:end_to_end.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/api/v0:compile_mlir_util_test PASSED in 5.2s //tensorflow/compiler/mlir/tf2xla/api/v0:compile_tf_graph_test PASSED in 0.2s //tensorflow/compiler/mlir/tf2xla/api/v1:legalize_tf_test PASSED in 23.3s //tensorflow/compiler/mlir/tf2xla/internal:mlir_pass_instrumentation_test PASSED in 6.1s //tensorflow/compiler/mlir/tf2xla/tests:adjust-layout.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_runtime_pipeline.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_sparsification.mlir.test PASSED in 11.2s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-BatchMatMulV2.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-binary-elementwise.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-collective.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-communication.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-include-tf2xla-fallback.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-no-tf2xla-fallback.mlir.test PASSED in 5.8s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-prefer-tf2xla.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-types.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-with-tf2xla-hlo-importer.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf.mlir.test PASSED in 8.4s //tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_cpu.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_gpu.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization-no-chlo.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/transforms:legalization_op_config_test PASSED in 29.3s //tensorflow/compiler/mlir/tf2xla/transforms:legalize_tf_types_test PASSED in 14.4s //tensorflow/compiler/mlir/tf2xla/transforms:tf2xla_rewriter_test PASSED in 19.9s //tensorflow/compiler/mlir/tf2xla/transforms:verify_tfxla_legalization_test PASSED in 17.6s //tensorflow/compiler/mlir/tf2xla/transforms:xla_legalize_targets_test PASSED in 0.6s //tensorflow/compiler/mlir/tf2xla/transforms:xla_legalize_tf_test PASSED in 3.3s //tensorflow/compiler/mlir/tfr:graph_decompose_test PASSED in 29.9s //tensorflow/compiler/mlir/tfr:node_expansion_test PASSED in 10.5s //tensorflow/compiler/mlir/tfr:op_reg_gen_test PASSED in 22.9s //tensorflow/compiler/mlir/tfr:tfr_decompose_ctx_test PASSED in 6.8s //tensorflow/compiler/mlir/tfr:tfr_gen_test PASSED in 27.1s //tensorflow/compiler/mlir/tfr/examples/customization:test_ops_test PASSED in 30.1s //tensorflow/compiler/mlir/tfr/examples/mnist:mnist_ops_test PASSED in 28.0s //tensorflow/compiler/mlir/tfr/examples/pad:pad_ops_test PASSED in 28.6s //tensorflow/compiler/mlir/tfrt/tests:batch_function_fallback_resource_variable_as_captured_tensor.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tfrt/tests:batch_function_lowering.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests:convert_ref_variables.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tfrt/tests:cross_device_transfer.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests:deduplicate_if_results.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests:fuse_tpu_compile_and_execute_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests:hoist_invariant_ops.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tfrt/tests:hoist_invariant_ops_mlrt.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests:optimize.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tfrt/tests:remove_device_attribute.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tfrt/tests:sink_in_invariant_ops.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tfrt/tests:xla_launch_fallback.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests:xla_rewrite.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/analysis:cost_analysis.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tfrt/tests/analysis:tensor_array_side_effect_analysis.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/analysis:update_op_cost_in_tfrt_mlir_test PASSED in 1.1s //tensorflow/compiler/mlir/tfrt/tests/ir:fallback_opt.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/ir:tfrt_fallback_util_test PASSED in 0.5s //tensorflow/compiler/mlir/tfrt/tests/mlrt:assign_op_key.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tfrt/tests/mlrt:fuse_mlrt_ops.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tfrt/tests/mlrt:inline.mlir.test PASSED in 17.6s //tensorflow/compiler/mlir/tfrt/tests/mlrt:parallelization.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tfrt/tests/mlrt:tf_to_mlrt.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tfrt/tests/mlrt:tpu_conversions.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tfrt/tests/mlrt:while_to_map_fn.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:attributes.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:basic.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:batch_function_deduplicate.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:batch_function_deduplicate_failed.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:const_tensor.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:control_flow.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:decompose_resource_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:derived_attrs.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:device_conversion.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:errors.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback_canonicalization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:fallback_inline.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_attributes.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_attributes_multiple_callers.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:func_use_fallback_tensor.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:insert_fallback_tensor_copy.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:merge_tf_if_ops.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:optimize_tf_control_flow_side_effect.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:remove_tf_if_const_args.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:reorder_assert.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:side_effects.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:tf_to_corert_pipeline.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:tf_to_corert_pipeline_refvar.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tfrt/tests/tf_to_corert:whileop.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tfrt/translate/mlrt:mlir_to_bytecode_test PASSED in 0.2s //tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_deallocation.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_reuse.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:bufferize.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tools/kernel_gen/tests:copy_cleanup.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tools/kernel_gen/tests:embed_tf_framework.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tools/kernel_gen/tests:invalid.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:isinf.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:ops.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/tools/kernel_gen/tests:parallel_loops_to_sequential.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:rewrite_tf_framework_assert.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tanh.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf-legalize-to-lmhlo.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_abi_knowledge.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_framework_legalize_to_llvm.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_kernel_gpu_launch_to_llvm.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_to_jit_invocations.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tosa/tests:convert-tfl-uint8.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tosa/tests:convert_metadata.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:fuse-bias-tf.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:lower-complex-types.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tosa/tests:lower_global_tensors.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tosa/tests:multi_add.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tosa/tests:retain_call_once_funcs.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tosa/tests:strip-quant-types.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tosa/tests:strip_metadata.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tosa/tests:tf-tfl-to-tosa-pipeline.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tosa/tests:tf-to-tosa-pipeline.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-dequantize_softmax.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline-filtered.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline.mlir.test PASSED in 6.3s //tensorflow/compiler/mlir/tosa/tests:verify_fully_converted.mlir.test PASSED in 0.6s //tensorflow/compiler/tests:adadelta_test_cpu PASSED in 18.6s //tensorflow/compiler/tests:adagrad_da_test_cpu PASSED in 13.9s //tensorflow/compiler/tests:adagrad_test_cpu PASSED in 17.7s //tensorflow/compiler/tests:adam_test_cpu PASSED in 13.7s //tensorflow/compiler/tests:add_n_test_cpu PASSED in 26.9s //tensorflow/compiler/tests:argminmax_test_cpu PASSED in 14.1s //tensorflow/compiler/tests:argminmax_test_cpu_mlir_bridge_test PASSED in 15.1s //tensorflow/compiler/tests:bucketize_op_test_cpu PASSED in 10.3s //tensorflow/compiler/tests:bucketize_op_test_cpu_mlir_bridge_test PASSED in 10.3s //tensorflow/compiler/tests:case_test_cpu PASSED in 10.4s //tensorflow/compiler/tests:cast_ops_test_cpu PASSED in 9.6s //tensorflow/compiler/tests:cast_ops_test_cpu_mlir_bridge_test PASSED in 8.5s //tensorflow/compiler/tests:categorical_op_test_cpu PASSED in 29.8s //tensorflow/compiler/tests:categorical_op_test_cpu_mlir_bridge_test PASSED in 13.9s //tensorflow/compiler/tests:cholesky_op_test_cpu PASSED in 20.4s //tensorflow/compiler/tests:cholesky_op_test_cpu_mlir_bridge_test PASSED in 20.7s //tensorflow/compiler/tests:clustering_test_cpu PASSED in 10.9s //tensorflow/compiler/tests:clustering_test_cpu_mlir_bridge_test PASSED in 11.8s //tensorflow/compiler/tests:concat_ops_test_cpu PASSED in 10.5s //tensorflow/compiler/tests:concat_ops_test_cpu_mlir_bridge_test PASSED in 12.2s //tensorflow/compiler/tests:cond_test_cpu PASSED in 11.6s //tensorflow/compiler/tests:const_arg_test_cpu PASSED in 11.2s //tensorflow/compiler/tests:const_test_cpu PASSED in 10.7s //tensorflow/compiler/tests:data_format_ops_test_cpu PASSED in 12.1s //tensorflow/compiler/tests:data_format_ops_test_cpu_mlir_bridge_test PASSED in 18.4s //tensorflow/compiler/tests:dense_layer_test_cpu PASSED in 17.8s //tensorflow/compiler/tests:dynamic_slice_ops_test_cpu PASSED in 14.8s //tensorflow/compiler/tests:dynamic_slice_ops_test_cpu_mlir_bridge_test PASSED in 13.2s //tensorflow/compiler/tests:dynamic_stitch_test_cpu PASSED in 10.2s //tensorflow/compiler/tests:dynamic_stitch_test_cpu_mlir_bridge_test PASSED in 10.2s //tensorflow/compiler/tests:eager_test_cpu PASSED in 36.7s //tensorflow/compiler/tests:einsum_op_test_cpu PASSED in 11.4s //tensorflow/compiler/tests:einsum_op_test_cpu_mlir_bridge_test PASSED in 10.7s //tensorflow/compiler/tests:ensure_shape_op_test_cpu PASSED in 9.1s //tensorflow/compiler/tests:extract_image_patches_op_test_cpu PASSED in 12.0s //tensorflow/compiler/tests:extract_image_patches_op_test_cpu_mlir_bridge_test PASSED in 11.0s //tensorflow/compiler/tests:fake_quant_ops_test_cpu PASSED in 18.5s //tensorflow/compiler/tests:fake_quant_ops_test_cpu_mlir_bridge_test PASSED in 27.7s //tensorflow/compiler/tests:fifo_queue_test_cpu PASSED in 10.7s //tensorflow/compiler/tests:fifo_queue_test_cpu_mlir_bridge_test PASSED in 27.5s //tensorflow/compiler/tests:ftrl_ops_test_cpu PASSED in 11.3s //tensorflow/compiler/tests:ftrl_ops_test_cpu_mlir_bridge_test PASSED in 29.7s //tensorflow/compiler/tests:function_test_cpu PASSED in 9.2s //tensorflow/compiler/tests:function_test_cpu_mlir_bridge_test PASSED in 9.4s //tensorflow/compiler/tests:gather_nd_op_test_cpu PASSED in 16.5s //tensorflow/compiler/tests:gather_nd_op_test_cpu_mlir_bridge_test PASSED in 27.4s //tensorflow/compiler/tests:gather_test_cpu PASSED in 39.9s //tensorflow/compiler/tests:gather_test_cpu_mlir_bridge_test PASSED in 44.1s //tensorflow/compiler/tests:jit_test_cpu PASSED in 40.4s //tensorflow/compiler/tests:listdiff_op_test_cpu PASSED in 11.2s //tensorflow/compiler/tests:listdiff_op_test_cpu_mlir_bridge_test PASSED in 20.7s //tensorflow/compiler/tests:lrn_ops_test_cpu PASSED in 11.2s //tensorflow/compiler/tests:lrn_ops_test_cpu_mlir_bridge_test PASSED in 9.4s //tensorflow/compiler/tests:lstm_test_cpu PASSED in 24.1s //tensorflow/compiler/tests:manip_ops_test_cpu PASSED in 28.9s //tensorflow/compiler/tests:manip_ops_test_cpu_mlir_bridge_test PASSED in 15.9s //tensorflow/compiler/tests:matrix_band_part_test_cpu PASSED in 43.1s //tensorflow/compiler/tests:matrix_band_part_test_cpu_mlir_bridge_test PASSED in 38.9s //tensorflow/compiler/tests:matrix_inverse_op_test_cpu PASSED in 20.1s //tensorflow/compiler/tests:matrix_inverse_op_test_cpu_mlir_bridge_test PASSED in 23.9s //tensorflow/compiler/tests:matrix_solve_op_test_cpu PASSED in 18.9s //tensorflow/compiler/tests:matrix_solve_op_test_cpu_mlir_bridge_test PASSED in 19.9s //tensorflow/compiler/tests:matrix_triangular_solve_op_test_cpu PASSED in 23.0s //tensorflow/compiler/tests:matrix_triangular_solve_op_test_cpu_mlir_bridge_test PASSED in 38.1s //tensorflow/compiler/tests:momentum_test_cpu PASSED in 13.3s //tensorflow/compiler/tests:nary_ops_test_cpu PASSED in 12.5s //tensorflow/compiler/tests:nary_ops_test_cpu_mlir_bridge_test PASSED in 11.0s //tensorflow/compiler/tests:nullary_ops_test_cpu PASSED in 10.6s //tensorflow/compiler/tests:nullary_ops_test_cpu_mlir_bridge_test PASSED in 11.1s //tensorflow/compiler/tests:placeholder_test_cpu PASSED in 14.0s //tensorflow/compiler/tests:placeholder_test_cpu_mlir_bridge_test PASSED in 9.3s //tensorflow/compiler/tests:proximal_adagrad_test_cpu PASSED in 10.6s //tensorflow/compiler/tests:proximal_gradient_descent_test_cpu PASSED in 11.0s //tensorflow/compiler/tests:quantized_ops_test_cpu PASSED in 12.2s //tensorflow/compiler/tests:reduce_window_test_cpu PASSED in 10.9s //tensorflow/compiler/tests:reduce_window_test_cpu_mlir_bridge_test PASSED in 9.4s //tensorflow/compiler/tests:reshape_op_test_cpu PASSED in 11.3s //tensorflow/compiler/tests:reshape_op_test_cpu_mlir_bridge_test PASSED in 10.0s //tensorflow/compiler/tests:reverse_ops_test_cpu PASSED in 14.5s //tensorflow/compiler/tests:reverse_ops_test_cpu_mlir_bridge_test PASSED in 16.2s //tensorflow/compiler/tests:reverse_sequence_op_test_cpu PASSED in 11.3s //tensorflow/compiler/tests:reverse_sequence_op_test_cpu_mlir_bridge_test PASSED in 11.9s //tensorflow/compiler/tests:rmsprop_test_cpu PASSED in 10.8s //tensorflow/compiler/tests:scatter_nd_op_test_cpu PASSED in 51.6s //tensorflow/compiler/tests:scatter_nd_op_test_cpu_mlir_bridge_test PASSED in 26.4s //tensorflow/compiler/tests:searchsorted_op_test_cpu PASSED in 14.2s //tensorflow/compiler/tests:searchsorted_op_test_cpu_mlir_bridge_test PASSED in 12.8s //tensorflow/compiler/tests:segment_reduction_ops_test_cpu PASSED in 29.3s //tensorflow/compiler/tests:segment_reduction_ops_test_cpu_mlir_bridge_test PASSED in 26.8s //tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu PASSED in 17.5s //tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu_mlir_bridge_test PASSED in 20.7s //tensorflow/compiler/tests:slice_ops_test_cpu PASSED in 28.7s //tensorflow/compiler/tests:slice_ops_test_cpu_mlir_bridge_test PASSED in 25.0s //tensorflow/compiler/tests:sparse_to_dense_op_test_cpu PASSED in 11.4s //tensorflow/compiler/tests:sparse_to_dense_op_test_cpu_mlir_bridge_test PASSED in 11.5s //tensorflow/compiler/tests:stack_ops_test_cpu PASSED in 10.2s //tensorflow/compiler/tests:tensor_float_32_test_cpu PASSED in 30.0s //tensorflow/compiler/tests:tensor_float_32_test_cpu_mlir_bridge_test PASSED in 13.0s //tensorflow/compiler/tests:tensor_list_ops_test_cpu PASSED in 12.6s //tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu PASSED in 16.4s //tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu_mlir_bridge_test PASSED in 15.6s //tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu PASSED in 17.4s //tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu_mlir_bridge_test PASSED in 15.4s //tensorflow/compiler/tests:unique_ops_test_cpu PASSED in 10.3s //tensorflow/compiler/tests:variable_ops_test_cpu PASSED in 30.9s //tensorflow/compiler/tests:variable_ops_test_cpu_mlir_bridge_test PASSED in 17.0s //tensorflow/compiler/tests:where_op_test_cpu PASSED in 11.8s //tensorflow/compiler/tests:while_test_cpu PASSED in 12.8s //tensorflow/compiler/tests:xla_call_module_no_platform_check_test_cpu PASSED in 12.3s //tensorflow/compiler/tests:xla_call_module_no_shape_assertions_check_test_cpu PASSED in 13.5s //tensorflow/compiler/tests:xla_call_module_test_cpu PASSED in 18.6s //tensorflow/compiler/tests:xla_custom_call_ops_test_cpu PASSED in 25.1s //tensorflow/compiler/tests:xla_device_gpu_test_cpu PASSED in 10.2s //tensorflow/compiler/tests:xla_device_test_cpu PASSED in 21.4s //tensorflow/compiler/tests:xla_device_test_cpu_mlir_bridge_test PASSED in 20.6s //tensorflow/compiler/tests:xla_ops_test_cpu PASSED in 60.8s //tensorflow/compiler/tests:xla_ops_test_cpu_mlir_bridge_test PASSED in 37.8s //tensorflow/compiler/tests:xla_test_test PASSED in 9.0s //tensorflow/compiler/tf2xla:const_analysis_test PASSED in 6.1s //tensorflow/compiler/tf2xla:cpu_function_runtime_test PASSED in 0.2s //tensorflow/compiler/tf2xla:functionalize_cond_test PASSED in 1.5s //tensorflow/compiler/tf2xla:functionalize_control_flow_test PASSED in 0.8s //tensorflow/compiler/tf2xla:fused_batchnorm_reserve_space_test_cpu PASSED in 32.1s //tensorflow/compiler/tf2xla:graph_compiler_test PASSED in 5.0s //tensorflow/compiler/tf2xla:literal_util_test PASSED in 0.7s //tensorflow/compiler/tf2xla:resource_operation_table_test PASSED in 5.7s //tensorflow/compiler/tf2xla:resource_util_test_cpu PASSED in 2.4s //tensorflow/compiler/tf2xla:sharding_util_test PASSED in 0.9s //tensorflow/compiler/tf2xla:tf2xla_opset_test PASSED in 9.9s //tensorflow/compiler/tf2xla:tf2xla_test PASSED in 18.8s //tensorflow/compiler/tf2xla:tf2xla_util_test PASSED in 0.5s //tensorflow/compiler/tf2xla:xla_compiler_test PASSED in 15.3s //tensorflow/compiler/tf2xla:xla_jit_compiled_cpu_function_test PASSED in 17.8s //tensorflow/compiler/tf2xla:xla_op_registry_test PASSED in 4.6s //tensorflow/compiler/tf2xla/kernels:rng_converter_utils_test PASSED in 2.3s //tensorflow/compiler/xla:array2d_test PASSED in 0.2s //tensorflow/compiler/xla:array3d_test PASSED in 0.1s //tensorflow/compiler/xla:array4d_test PASSED in 0.4s //tensorflow/compiler/xla:array_test PASSED in 0.1s //tensorflow/compiler/xla:bit_cast_test PASSED in 0.3s //tensorflow/compiler/xla:comparison_util_test PASSED in 0.1s //tensorflow/compiler/xla:debug_options_parsers_test PASSED in 0.2s //tensorflow/compiler/xla:index_util_test PASSED in 0.1s //tensorflow/compiler/xla:iterator_util_test PASSED in 0.3s //tensorflow/compiler/xla:layout_test PASSED in 0.1s //tensorflow/compiler/xla:layout_util_test PASSED in 0.2s //tensorflow/compiler/xla:literal_test PASSED in 0.2s //tensorflow/compiler/xla:parse_flags_from_env_test PASSED in 0.6s //tensorflow/compiler/xla:permutation_util_test PASSED in 0.2s //tensorflow/compiler/xla:primitive_util_test PASSED in 0.1s //tensorflow/compiler/xla:refcounting_hash_map_test PASSED in 0.1s //tensorflow/compiler/xla:reference_util_test PASSED in 0.3s //tensorflow/compiler/xla:shape_test PASSED in 0.1s //tensorflow/compiler/xla:shape_tree_test PASSED in 0.9s //tensorflow/compiler/xla:shape_util_test PASSED in 2.0s //tensorflow/compiler/xla:status_macros_test PASSED in 0.1s //tensorflow/compiler/xla:text_literal_reader_test PASSED in 0.2s //tensorflow/compiler/xla:text_literal_writer_test PASSED in 0.1s //tensorflow/compiler/xla:types_test PASSED in 0.2s //tensorflow/compiler/xla:util_test PASSED in 0.1s //tensorflow/compiler/xla:window_util_test PASSED in 0.1s //tensorflow/compiler/xla/client:padding_test PASSED in 0.1s //tensorflow/compiler/xla/client:xla_builder_test PASSED in 0.3s //tensorflow/compiler/xla/client/lib:arithmetic_test_cpu PASSED in 8.7s //tensorflow/compiler/xla/client/lib:comparators_test_cpu PASSED in 7.0s //tensorflow/compiler/xla/client/lib:constants_test_cpu PASSED in 6.5s //tensorflow/compiler/xla/client/lib:logdet_test_cpu PASSED in 9.2s //tensorflow/compiler/xla/client/lib:math_test_cpu PASSED in 13.8s //tensorflow/compiler/xla/client/lib:matrix_test_cpu PASSED in 10.6s //tensorflow/compiler/xla/client/lib:pooling_test_cpu PASSED in 9.7s //tensorflow/compiler/xla/client/lib:qr_test_cpu PASSED in 11.2s //tensorflow/compiler/xla/client/lib:slicing_test_cpu PASSED in 6.9s //tensorflow/compiler/xla/client/lib:sorting_test_cpu PASSED in 11.0s //tensorflow/compiler/xla/examples/axpy:stablehlo_compile_test PASSED in 9.3s //tensorflow/compiler/xla/experiments/sm_bandwidth_benchmark:sm_bw_test PASSED in 0.5s //tensorflow/compiler/xla/hlo/evaluator:hlo_evaluator_test PASSED in 19.3s //tensorflow/compiler/xla/hlo/experimental/auto_sharding:auto_sharding_solver_test PASSED in 0.8s //tensorflow/compiler/xla/hlo/experimental/auto_sharding:auto_sharding_test PASSED in 5.7s //tensorflow/compiler/xla/hlo/transforms:hlo_constant_splitter_test PASSED in 1.5s //tensorflow/compiler/xla/hlo/utils:hlo_live_range_test PASSED in 0.7s //tensorflow/compiler/xla/hlo/utils:hlo_matchers_test PASSED in 1.3s //tensorflow/compiler/xla/hlo/utils:hlo_sharding_util_test PASSED in 0.2s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:collective_ops.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:fft.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:legalize_i1_vector_transfers.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:library_ops_to_cpu_runtime.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:lmhlo_custom_call.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:remove_copies_to_out_params.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:rng_bit_generator.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_abi_legalization.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_infeed.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_memref_element_cast_to_llvm.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_outfeed.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:add_concurrent_regions.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:add_hlo_trace.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_launch.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memcpy.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memset.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_case.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_custom_call.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_fft.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cholesky.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_conv.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cublas_lt_matmul.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_gemm.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_infeed.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_outfeed.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_send_recv.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_while.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:memref_get_global_to_arg.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:outline_cuda_graphs.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:stream_assignment.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/framework/tests:legalize-xla-framework.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir/framework/tests:outline-with-xla-framework.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir/framework/tests:xla-framework.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/math/transforms/tests:math_optimization.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir/memref/transforms/tests:aligned_allocations.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/runtime/ir/tests:ops.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/runtime/ir/tests:ops_verify.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/runtime/ir/tests:testlib.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/runtime/transforms:calling_convention_test PASSED in 0.5s //tensorflow/compiler/xla/mlir/runtime/transforms:type_converter_test PASSED in 0.3s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:compilation_pipeline.mlir.test PASSED in 4.0s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_asserts.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_custom_calls.mlir.test PASSED in 1.8s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:export_functions.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:ordinal_assignment.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:rt_to_llvm.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:erase-op-without-results.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:inline-scf-while.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:reduce-scf-forall-bounds.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-constant.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-value.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-operand-with-constant.mlir.test PASSED in 3.9s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:return-operands-of-terminator-operands.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:truncate-function.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:bisect.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:no-bug.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:snapshot.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir/tools/mlir_replay/public:execution_trace_utils_test PASSED in 0.2s //tensorflow/compiler/xla/mlir/utils:error_util_test PASSED in 0.1s //tensorflow/compiler/xla/mlir/xla_cpu/tests:bufferize.mlir.test PASSED in 1.8s //tensorflow/compiler/xla/mlir/xla_cpu/tests:invalid.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/xla_cpu/tests:ops.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/bufferization/hlo_one_shot_bufferize.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_hlo_broadcasts.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_hlo_no_broadcasts.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_mhlo.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/sparse_chlo_legalize_to_linalg.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/analysis.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/buffer_reuse.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/convert_deallocation_ops_to_llvm.mlir.test PASSED in 11.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocate.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocate_invalid.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_simplification.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_to_scf.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/split_alloc_tensors.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/add_debug_info.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/bufferization.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/collapse-shape.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/collect_stats.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/compose_extract_insert_slice.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/batch_matmul.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/conv_2d_nhwc_hwcf.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/dot.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/duplicate_fusions.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fibonacci.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fusion_outlining.mlir.test PASSED in 3.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fusion_planning_for_cpu.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/inline_fusion_clusters.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_bcast_map.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_matmul.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_reduce_map.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_reshape_map.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/matmul.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_1d.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_1d_map.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_2d.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_window.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reverse.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/scatter.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/sort.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/transpose.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/greedy_fusion.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/invalid.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/lower_vectors.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/nested_tiling_softmax.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/ops.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/optimize_linalg_ops.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/rewrite_forall_to_for.mlir.test PASSED in 1.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/simplify_dead_copy.mlir.test PASSED in 2.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/tile_by_one.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/tiling_softmax.mlir.test PASSED in 2.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/vectorize_copy.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/vectorize_for_cpu.mlir.test PASSED in 1.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-select-and-scatter.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-affine.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-gpu.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-parallel-loops.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-tensor-op.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/ops.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo_gpu/lhlo_gpu_ops.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/attrs.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/broadcast_propagation.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/bitcast.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/canonicalize.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/concatenate.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/convert.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/convolution.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/custom_call.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/folder_limit.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reduce.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reshape.mlir.test PASSED in 1.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reverse.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/scatter.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/transpose.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/tuple.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/while.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/constraint_fusion.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/convert_to_signless.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/expand_hlo_tuples.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/expand_ops_simplifier.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/group_reduction_dimensions.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-collapse-elementwise-map.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-einsum-to-dot-general.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-gather-to-torch-index-select.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-rng-to-linalg.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-shape-ops-to-standard.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-sort.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-arithmetic.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo-only-dynamic.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo-unranked.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-linalg.mlir.test PASSED in 2.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-memref-unranked.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-memref.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-stablehlo-experimental.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-stablehlo.mlir.test PASSED in 11.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-torch-index-select-to-gather.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/inlining.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/invalid.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-control-flow.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-hlo-shape-computations.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-mhlo-to-thlo.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-to-std.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/lower-complex.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/lower-general-dot.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/materialize-broadcasts.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/merge_assuming_ops.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_bytecode_customizations.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_dot.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_gather.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_reduction.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_scatter.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_flatten_tuple.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_infer_shape_type_methods.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_ops_prettyprint.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_reduce_pretty_print.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/ops.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/optimize-hlo.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/prepare-for-export.mlir.test PASSED in 1.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/reify-result-types.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/restrict_max_rank.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/shape_legalize_to_hlo.mlir.test PASSED in 1.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/shape_reification.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sink-constants-to-control-flow.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_gendot_lower.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_lower.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_rewriting.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_transpose.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/stablehlo-legalize-to-hlo.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/symbolic-shape-optimization.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/unfuse_batch_norm.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_bounds.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_conv_op.mlir.test PASSED in 1.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_reduce_op.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_reduce_window_op.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_scatter_op.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_select_and_scatter_op.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_while_op.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/while_prettyprint.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/bufferize.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/canonicalize.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/invalid.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/legalize_sort.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/ops.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/tiling.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:alloc_to_arg.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:assuming-structural-propagation.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tests:buffer_packing.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:bufferize.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:bufferize_one_shot.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:collapse_parallel_loops_to_1d_pass.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:detensorize_scf_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:index_type_llvm_lowering.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:legalize-trigonometric-to-approximation.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:lower_index_cast.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:propagate_static_shapes.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:rank-specialization.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:scalarization.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:shape-component-analysis.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:shape_simplification.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:test_userange.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:tile_loops.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:unbufferize.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:unroll-loops.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tools/mlir_interpreter/framework/tests:interpreter_value_test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tools/mlir_interpreter/framework/tests:tensor_or_memref_test PASSED in 0.2s //tensorflow/compiler/xla/pjrt:host_callback_test PASSED in 0.2s //tensorflow/compiler/xla/pjrt:lru_cache_test PASSED in 0.7s //tensorflow/compiler/xla/pjrt:pjrt_api_test PASSED in 0.4s //tensorflow/compiler/xla/pjrt:pjrt_client_test_cpu PASSED in 9.1s //tensorflow/compiler/xla/pjrt:pjrt_compiler_test PASSED in 0.2s //tensorflow/compiler/xla/pjrt:pjrt_executable_test PASSED in 0.2s //tensorflow/compiler/xla/pjrt:pjrt_stream_executor_client_test PASSED in 7.3s //tensorflow/compiler/xla/pjrt:semaphore_test PASSED in 0.1s //tensorflow/compiler/xla/pjrt:tf_pjrt_client_test PASSED in 8.9s //tensorflow/compiler/xla/pjrt:tfrt_cpu_pjrt_client_test PASSED in 9.0s //tensorflow/compiler/xla/pjrt:tracked_device_buffer_test PASSED in 7.7s //tensorflow/compiler/xla/pjrt:tracked_tfrt_cpu_device_buffer_test PASSED in 0.1s //tensorflow/compiler/xla/pjrt:transpose_test PASSED in 41.4s //tensorflow/compiler/xla/pjrt/c:pjrt_c_api_cpu_test PASSED in 8.7s //tensorflow/compiler/xla/pjrt/c:pjrt_c_api_helpers_test PASSED in 1.2s //tensorflow/compiler/xla/pjrt/distributed:client_server_test PASSED in 57.7s //tensorflow/compiler/xla/pjrt/distributed:topology_util_test PASSED in 0.1s //tensorflow/compiler/xla/python:outfeed_receiver_test_cpu PASSED in 7.2s //tensorflow/compiler/xla/python:xplane_to_profile_instructions_test PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:array_test PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:array_test_no_impl PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:client_test_no_impl PASSED in 0.4s //tensorflow/compiler/xla/python/ifrt:future_test PASSED in 0.2s //tensorflow/compiler/xla/python/ifrt:index_domain_test PASSED in 1.4s //tensorflow/compiler/xla/python/ifrt:index_test PASSED in 0.2s //tensorflow/compiler/xla/python/ifrt:serdes_test PASSED in 0.1s //tensorflow/compiler/xla/python/ifrt:shape_test PASSED in 0.4s //tensorflow/compiler/xla/python/ifrt:sharding_serdes_test PASSED in 0.4s //tensorflow/compiler/xla/python/ifrt:sharding_test PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:tuple_test_no_impl PASSED in 0.7s //tensorflow/compiler/xla/python/ifrt/ir/tests:executable_test_no_impl PASSED in 3.3s //tensorflow/compiler/xla/python/ifrt/ir/tests:ifrt_duplicated_callee_elimination.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/python/ifrt/ir/tests:spmd_interface_verification.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_array.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_assemble.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_attrs.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_call.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_call_loaded_executable.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_disassemble.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_loaded_executable.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_reshard.mlir.test PASSED in 1.6s //tensorflow/compiler/xla/python/ifrt/support:sharding_param_to_op_sharding_test PASSED in 0.4s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_array_impl_test_tfrt_cpu PASSED in 14.2s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_client_impl_test_tfrt_cpu PASSED in 8.0s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_executable_impl_test_tfrt_cpu PASSED in 7.7s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_tuple_impl_test_tfrt_cpu PASSED in 7.9s //tensorflow/compiler/xla/python/pjrt_ifrt:xla_executable_test_no_impl PASSED in 2.8s //tensorflow/compiler/xla/python/pjrt_ifrt:xla_program_serdes_test PASSED in 1.7s //tensorflow/compiler/xla/python/pjrt_ifrt:xla_sharding_test PASSED in 7.2s //tensorflow/compiler/xla/python_api:xla_literal_test PASSED in 1.6s //tensorflow/compiler/xla/python_api:xla_shape_test PASSED in 1.5s //tensorflow/compiler/xla/rpc:grpc_client_test PASSED in 2.3s //tensorflow/compiler/xla/runtime:arguments_test PASSED in 0.1s //tensorflow/compiler/xla/runtime:async_runtime_test PASSED in 0.1s //tensorflow/compiler/xla/runtime:custom_call_test PASSED in 1.9s //tensorflow/compiler/xla/runtime:diagnostics_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:executable_test PASSED in 3.6s //tensorflow/compiler/xla/runtime:ffi_test PASSED in 1.1s //tensorflow/compiler/xla/runtime:map_by_type_test PASSED in 0.1s //tensorflow/compiler/xla/runtime:module_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:results_test PASSED in 0.3s //tensorflow/compiler/xla/runtime:state_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:symbolic_shape_test PASSED in 0.5s //tensorflow/compiler/xla/runtime:type_id_test PASSED in 0.1s //tensorflow/compiler/xla/service:algebraic_simplifier_overflow_test_cpu PASSED in 7.0s //tensorflow/compiler/xla/service:algebraic_simplifier_test PASSED in 9.3s //tensorflow/compiler/xla/service:all_gather_broadcast_reorder_test PASSED in 0.9s //tensorflow/compiler/xla/service:all_gather_combiner_test PASSED in 2.0s //tensorflow/compiler/xla/service:all_gather_decomposer_test PASSED in 1.6s //tensorflow/compiler/xla/service:all_reduce_combiner_test PASSED in 0.8s //tensorflow/compiler/xla/service:all_reduce_contiguous_test PASSED in 0.8s //tensorflow/compiler/xla/service:all_reduce_folder_test PASSED in 0.7s //tensorflow/compiler/xla/service:all_reduce_promotion_test PASSED in 0.8s //tensorflow/compiler/xla/service:all_reduce_reassociate_test PASSED in 0.7s //tensorflow/compiler/xla/service:all_reduce_simplifier_test PASSED in 0.7s //tensorflow/compiler/xla/service:ar_crs_combiner_test PASSED in 0.7s //tensorflow/compiler/xla/service:async_collective_creator_test PASSED in 0.8s //tensorflow/compiler/xla/service:async_op_canonicalizer_test PASSED in 1.2s //tensorflow/compiler/xla/service:batch_dot_simplification_test PASSED in 2.4s //tensorflow/compiler/xla/service:batchnorm_expander_test_cpu PASSED in 6.0s //tensorflow/compiler/xla/service:bfloat16_conversion_folding_test PASSED in 0.8s //tensorflow/compiler/xla/service:bfloat16_propagation_test PASSED in 1.0s //tensorflow/compiler/xla/service:bitcast_dtypes_expander_test PASSED in 1.5s //tensorflow/compiler/xla/service:broadcast_canonicalizer_test PASSED in 1.4s //tensorflow/compiler/xla/service:buffer_assignment_test PASSED in 10.5s //tensorflow/compiler/xla/service:call_graph_test PASSED in 1.5s //tensorflow/compiler/xla/service:call_inliner_test PASSED in 0.8s //tensorflow/compiler/xla/service:change_op_data_type_test PASSED in 1.0s //tensorflow/compiler/xla/service:collective_ops_utils_test PASSED in 0.2s //tensorflow/compiler/xla/service:collective_permute_decomposer_test PASSED in 1.5s //tensorflow/compiler/xla/service:collective_pipeliner_test PASSED in 1.1s //tensorflow/compiler/xla/service:collective_transformation_reorderer_test PASSED in 2.1s //tensorflow/compiler/xla/service:collectives_schedule_linearizer_test PASSED in 0.8s //tensorflow/compiler/xla/service:compilation_environments_test PASSED in 0.2s //tensorflow/compiler/xla/service:conditional_canonicalizer_test PASSED in 0.8s //tensorflow/compiler/xla/service:conditional_code_motion_test PASSED in 2.3s //tensorflow/compiler/xla/service:conditional_simplifier_test PASSED in 2.2s //tensorflow/compiler/xla/service:conditional_to_select_test PASSED in 0.9s //tensorflow/compiler/xla/service:constant_value_test PASSED in 0.4s //tensorflow/compiler/xla/service:convert_async_collectives_to_sync_test PASSED in 3.8s //tensorflow/compiler/xla/service:convert_mover_test PASSED in 1.4s //tensorflow/compiler/xla/service:convert_operand_folding_test PASSED in 1.4s //tensorflow/compiler/xla/service:convolution_4d_expander_test PASSED in 0.6s //tensorflow/compiler/xla/service:convolution_group_converter_test PASSED in 0.9s //tensorflow/compiler/xla/service:convolution_pred_expander_test PASSED in 1.3s //tensorflow/compiler/xla/service:copy_insertion_test PASSED in 4.8s //tensorflow/compiler/xla/service:custom_call_status_test PASSED in 0.2s //tensorflow/compiler/xla/service:defuser_test PASSED in 2.0s //tensorflow/compiler/xla/service:despecializer_test PASSED in 2.2s //tensorflow/compiler/xla/service:dfs_hlo_visitor_with_default_test PASSED in 0.8s //tensorflow/compiler/xla/service:dot_decomposer_test PASSED in 0.7s //tensorflow/compiler/xla/service:dot_dimension_merger_test PASSED in 0.9s //tensorflow/compiler/xla/service:dot_merger_test PASSED in 3.3s //tensorflow/compiler/xla/service:dynamic_dimension_inference_test PASSED in 2.3s //tensorflow/compiler/xla/service:dynamic_dimension_simplifier_test PASSED in 1.3s //tensorflow/compiler/xla/service:dynamic_index_splitter_test PASSED in 1.3s //tensorflow/compiler/xla/service:dynamic_padder_test_cpu PASSED in 18.2s //tensorflow/compiler/xla/service:dynamic_parameter_binding_test PASSED in 2.2s //tensorflow/compiler/xla/service:dynamic_update_slice_test_cpu PASSED in 8.6s //tensorflow/compiler/xla/service:elemental_ir_emitter_test_cpu PASSED in 12.7s //tensorflow/compiler/xla/service:flatten_call_graph_test PASSED in 0.8s //tensorflow/compiler/xla/service:float_normalization_test PASSED in 1.7s //tensorflow/compiler/xla/service:fusion_node_indexing_evaluation_test PASSED in 1.5s //tensorflow/compiler/xla/service:gather_expander_test PASSED in 1.7s //tensorflow/compiler/xla/service:gather_simplifier_test PASSED in 1.0s //tensorflow/compiler/xla/service:heap_simulator_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_alias_analysis_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_casting_utils_test PASSED in 9.0s //tensorflow/compiler/xla/service:hlo_computation_deduplicator_test PASSED in 1.1s //tensorflow/compiler/xla/service:hlo_computation_test PASSED in 2.2s //tensorflow/compiler/xla/service:hlo_constant_folding_test PASSED in 4.5s //tensorflow/compiler/xla/service:hlo_cost_analysis_test PASSED in 7.4s //tensorflow/compiler/xla/service:hlo_creation_utils_test PASSED in 3.7s //tensorflow/compiler/xla/service:hlo_cse_test PASSED in 10.3s //tensorflow/compiler/xla/service:hlo_dataflow_analysis_test PASSED in 1.3s //tensorflow/compiler/xla/service:hlo_dce_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_domain_test PASSED in 0.9s //tensorflow/compiler/xla/service:hlo_element_type_converter_test PASSED in 0.7s //tensorflow/compiler/xla/service:hlo_execution_profile_test PASSED in 4.7s //tensorflow/compiler/xla/service:hlo_graph_dumper_test PASSED in 0.7s //tensorflow/compiler/xla/service:hlo_input_output_alias_config_test PASSED in 1.2s //tensorflow/compiler/xla/service:hlo_instruction_test PASSED in 1.3s //tensorflow/compiler/xla/service:hlo_liveness_analysis_test PASSED in 1.2s //tensorflow/compiler/xla/service:hlo_memory_scheduler_test PASSED in 1.4s //tensorflow/compiler/xla/service:hlo_module_dce_test PASSED in 0.7s //tensorflow/compiler/xla/service:hlo_module_metadata_test PASSED in 0.1s //tensorflow/compiler/xla/service:hlo_module_test PASSED in 1.5s //tensorflow/compiler/xla/service:hlo_opcode_test PASSED in 3.7s //tensorflow/compiler/xla/service:hlo_ordering_test PASSED in 1.0s //tensorflow/compiler/xla/service:hlo_parser_test PASSED in 1.5s //tensorflow/compiler/xla/service:hlo_pass_pipeline_test PASSED in 0.9s //tensorflow/compiler/xla/service:hlo_phi_graph_test PASSED in 1.4s //tensorflow/compiler/xla/service:hlo_proto_util_test PASSED in 1.0s //tensorflow/compiler/xla/service:hlo_reachability_test PASSED in 0.7s //tensorflow/compiler/xla/service:hlo_rematerialization_test PASSED in 1.1s //tensorflow/compiler/xla/service:hlo_rematerialization_test_utils_test PASSED in 1.0s //tensorflow/compiler/xla/service:hlo_replication_analysis_test PASSED in 3.7s //tensorflow/compiler/xla/service:hlo_schedule_test PASSED in 1.1s //tensorflow/compiler/xla/service:hlo_sharding_test PASSED in 3.1s //tensorflow/compiler/xla/service:hlo_value_semantics_analysis_test PASSED in 1.0s //tensorflow/compiler/xla/service:hlo_verifier_test PASSED in 1.5s //tensorflow/compiler/xla/service:indexed_array_analysis_test PASSED in 0.7s //tensorflow/compiler/xla/service:instruction_fusion_test PASSED in 1.0s //tensorflow/compiler/xla/service:latency_hiding_scheduler_test PASSED in 1.0s //tensorflow/compiler/xla/service:layout_assignment_test PASSED in 6.3s //tensorflow/compiler/xla/service:layout_normalization_test PASSED in 1.8s //tensorflow/compiler/xla/service:logistic_expander_test PASSED in 1.0s //tensorflow/compiler/xla/service:loop_schedule_linearizer_test PASSED in 0.9s //tensorflow/compiler/xla/service:map_inliner_test PASSED in 0.9s //tensorflow/compiler/xla/service:mapped_ptr_container_sorter_test PASSED in 0.5s //tensorflow/compiler/xla/service:memory_space_assignment_best_fit_repacker_test PASSED in 0.2s //tensorflow/compiler/xla/service:memory_space_assignment_test PASSED in 2.2s //tensorflow/compiler/xla/service:memory_space_propagation_test PASSED in 1.1s //tensorflow/compiler/xla/service:name_uniquer_test PASSED in 0.1s //tensorflow/compiler/xla/service:operand_upcaster_test PASSED in 1.0s //tensorflow/compiler/xla/service:optimize_input_output_buffer_alias_test PASSED in 4.4s //tensorflow/compiler/xla/service:pattern_matcher_gmock_test PASSED in 0.2s //tensorflow/compiler/xla/service:pattern_matcher_test PASSED in 0.7s //tensorflow/compiler/xla/service:profile_guided_latency_estimator_test PASSED in 0.7s //tensorflow/compiler/xla/service:real_imag_expander_test PASSED in 0.8s //tensorflow/compiler/xla/service:reduce_decomposer_test PASSED in 0.7s //tensorflow/compiler/xla/service:reduce_scatter_combiner_test PASSED in 0.9s //tensorflow/compiler/xla/service:reduce_scatter_decomposer_test PASSED in 2.0s //tensorflow/compiler/xla/service:reduce_scatter_reassociate_test PASSED in 0.9s //tensorflow/compiler/xla/service:reshape_decomposer_test PASSED in 0.7s //tensorflow/compiler/xla/service:reshape_mover_test PASSED in 0.7s //tensorflow/compiler/xla/service:result_caster_test PASSED in 0.9s //tensorflow/compiler/xla/service:root_instruction_sinker_test PASSED in 0.6s //tensorflow/compiler/xla/service:scatter_expander_test PASSED in 0.9s //tensorflow/compiler/xla/service:scatter_simplifier_test PASSED in 0.7s //tensorflow/compiler/xla/service:select_and_scatter_expander_test PASSED in 0.7s //tensorflow/compiler/xla/service:shape_inference_test PASSED in 0.3s //tensorflow/compiler/xla/service:shaped_buffer_test PASSED in 8.8s //tensorflow/compiler/xla/service:sharding_propagation_test PASSED in 3.8s //tensorflow/compiler/xla/service:sharding_remover_test PASSED in 0.7s //tensorflow/compiler/xla/service:simplify_fp_conversions_test PASSED in 1.0s //tensorflow/compiler/xla/service:slice_sinker_test PASSED in 1.5s //tensorflow/compiler/xla/service:sort_simplifier_test PASSED in 1.0s //tensorflow/compiler/xla/service:space_to_batch_converter_test PASSED in 1.1s //tensorflow/compiler/xla/service:stable_sort_expander_test PASSED in 0.7s //tensorflow/compiler/xla/service:stochastic_convert_decomposer_test PASSED in 1.2s //tensorflow/compiler/xla/service:stream_pool_test PASSED in 0.2s //tensorflow/compiler/xla/service:topk_rewriter_test PASSED in 5.9s //tensorflow/compiler/xla/service:transpose_folding_test PASSED in 1.4s //tensorflow/compiler/xla/service:tuple_points_to_analysis_test PASSED in 1.3s //tensorflow/compiler/xla/service:tuple_simplifier_test PASSED in 1.5s //tensorflow/compiler/xla/service:tuple_util_test PASSED in 0.7s //tensorflow/compiler/xla/service:value_range_test PASSED in 0.8s //tensorflow/compiler/xla/service:while_loop_all_reduce_code_motion_test PASSED in 1.2s //tensorflow/compiler/xla/service:while_loop_analysis_test PASSED in 1.1s //tensorflow/compiler/xla/service:while_loop_concat_code_motion_test PASSED in 0.8s //tensorflow/compiler/xla/service:while_loop_constant_sinking_test PASSED in 1.0s //tensorflow/compiler/xla/service:while_loop_expensive_invariant_code_motion_test PASSED in 1.7s //tensorflow/compiler/xla/service:while_loop_invariant_code_motion_test PASSED in 0.8s //tensorflow/compiler/xla/service:while_loop_simplifier_test PASSED in 1.6s //tensorflow/compiler/xla/service:while_loop_trip_count_annotator_test PASSED in 0.8s //tensorflow/compiler/xla/service:while_util_test PASSED in 0.6s //tensorflow/compiler/xla/service:xla_aot_compile_stablehlo_cpu_test PASSED in 9.8s //tensorflow/compiler/xla/service:xla_debug_info_manager_test PASSED in 0.8s //tensorflow/compiler/xla/service:zero_sized_hlo_elimination_test PASSED in 1.3s //tensorflow/compiler/xla/service/cpu:conv_canonicalization_test PASSED in 1.4s //tensorflow/compiler/xla/service/cpu:cpu_eigen_tensor_alignment_test PASSED in 1.8s //tensorflow/compiler/xla/service/cpu:cpu_instruction_fusion_test PASSED in 0.9s //tensorflow/compiler/xla/service/cpu:cpu_layout_assignment_test PASSED in 3.0s //tensorflow/compiler/xla/service/cpu:ir_emission_utils_test PASSED in 1.3s //tensorflow/compiler/xla/service/cpu:parallel_task_assignment_test PASSED in 3.1s //tensorflow/compiler/xla/service/cpu:runtime_fft_test PASSED in 0.4s //tensorflow/compiler/xla/service/cpu:shape_partition_test PASSED in 0.7s //tensorflow/compiler/xla/service/cpu:xfeed_manager_test PASSED in 0.6s //tensorflow/compiler/xla/service/cpu/tests:cpu_bytesizeof_test PASSED in 0.7s //tensorflow/compiler/xla/service/cpu/tests:cpu_dyn_shape_test PASSED in 7.4s //tensorflow/compiler/xla/service/cpu/tests:cpu_eigen_dot_operation_test PASSED in 8.5s //tensorflow/compiler/xla/service/cpu/tests:cpu_external_constants_test PASSED in 23.2s //tensorflow/compiler/xla/service/cpu/tests:cpu_fusion_test PASSED in 6.7s //tensorflow/compiler/xla/service/cpu/tests:cpu_infeed_test PASSED in 7.5s //tensorflow/compiler/xla/service/cpu/tests:cpu_intrinsic_test PASSED in 8.0s //tensorflow/compiler/xla/service/cpu/tests:cpu_key_value_sort_test PASSED in 8.1s //tensorflow/compiler/xla/service/cpu/tests:cpu_literal_caching_test PASSED in 7.2s //tensorflow/compiler/xla/service/cpu/tests:cpu_noalias_test PASSED in 7.0s //tensorflow/compiler/xla/service/cpu/tests:cpu_outfeed_test PASSED in 7.4s //tensorflow/compiler/xla/service/cpu/tests:cpu_profiling_test PASSED in 8.1s //tensorflow/compiler/xla/service/cpu/tests:cpu_spmd_compile_test PASSED in 6.2s //tensorflow/compiler/xla/service/cpu/tests:cpu_topk_test PASSED in 8.5s //tensorflow/compiler/xla/service/cpu/tests:cpu_vectorization_test PASSED in 10.7s //tensorflow/compiler/xla/service/cpu/tests:cpu_while_test PASSED in 8.7s //tensorflow/compiler/xla/service/cpu/tests:tree_reduction_rewriter_test PASSED in 8.0s //tensorflow/compiler/xla/service/gpu:alias_passthrough_params_test PASSED in 1.0s //tensorflow/compiler/xla/service/gpu:all_reduce_blueconnect_test PASSED in 0.7s //tensorflow/compiler/xla/service/gpu:autotuner_util_test PASSED in 0.3s //tensorflow/compiler/xla/service/gpu:backend_configs_test PASSED in 0.7s //tensorflow/compiler/xla/service/gpu:copy_fusion_test PASSED in 1.4s //tensorflow/compiler/xla/service/gpu:cublas_pad_for_gemms_test PASSED in 4.2s //tensorflow/compiler/xla/service/gpu:cudnn_pad_for_convolutions_test PASSED in 1.3s //tensorflow/compiler/xla/service/gpu:cudnn_simplify_padding_test PASSED in 1.2s //tensorflow/compiler/xla/service/gpu:cudnn_support_utils_test PASSED in 1.2s //tensorflow/compiler/xla/service/gpu:cudnn_vectorize_convolutions_test PASSED in 1.4s //tensorflow/compiler/xla/service/gpu:gemm_rewriter_triton_test PASSED in 1.7s //tensorflow/compiler/xla/service/gpu:gpu_async_collective_annotator_test PASSED in 0.7s //tensorflow/compiler/xla/service/gpu:gpu_conv_padding_legalization_test PASSED in 0.8s //tensorflow/compiler/xla/service/gpu:gpu_conv_rewriter_test PASSED in 1.2s //tensorflow/compiler/xla/service/gpu:gpu_convert_async_collectives_to_sync_test PASSED in 1.9s //tensorflow/compiler/xla/service/gpu:gpu_fusible_test PASSED in 1.7s //tensorflow/compiler/xla/service/gpu:gpu_hlo_cost_analysis_test PASSED in 8.9s //tensorflow/compiler/xla/service/gpu:gpu_performance_model_test PASSED in 2.1s //tensorflow/compiler/xla/service/gpu:gpu_sanitize_constant_names_test PASSED in 1.8s //tensorflow/compiler/xla/service/gpu:hlo_algorithm_denylist_test PASSED in 0.3s //tensorflow/compiler/xla/service/gpu:hlo_fusion_stats_test PASSED in 0.8s //tensorflow/compiler/xla/service/gpu:instruction_fusion_test PASSED in 1.3s //tensorflow/compiler/xla/service/gpu:ir_emission_utils_test PASSED in 1.6s //tensorflow/compiler/xla/service/gpu:matmul_utils_test PASSED in 1.0s //tensorflow/compiler/xla/service/gpu:move_copy_to_users_test PASSED in 0.9s //tensorflow/compiler/xla/service/gpu:multi_output_fusion_test PASSED in 1.8s //tensorflow/compiler/xla/service/gpu:non_atomically_upgradeable_rw_lock_test PASSED in 0.1s //tensorflow/compiler/xla/service/gpu:priority_fusion_test PASSED in 2.1s //tensorflow/compiler/xla/service/gpu:reduction_splitter_test PASSED in 1.3s //tensorflow/compiler/xla/service/gpu:scatter_slice_simplifier_test PASSED in 1.9s //tensorflow/compiler/xla/service/gpu:softmax_rewriter_triton_test PASSED in 1.9s //tensorflow/compiler/xla/service/gpu:target_util_test PASSED in 0.9s //tensorflow/compiler/xla/service/gpu:topk_splitter_test PASSED in 45.2s //tensorflow/compiler/xla/service/gpu:variadic_op_splitter_test PASSED in 1.3s //tensorflow/compiler/xla/service/gpu:while_transformer_test PASSED in 2.0s //tensorflow/compiler/xla/service/gpu/llvm_gpu_backend:utils_test PASSED in 0.5s //tensorflow/compiler/xla/service/gpu/tests:gpu_reduce_scatter_creator_test PASSED in 1.5s //tensorflow/compiler/xla/service/gpu/tests:reduction_degenerate_dim_remover_test PASSED in 1.5s //tensorflow/compiler/xla/service/gpu/tests:reduction_dimension_grouper_test PASSED in 1.2s //tensorflow/compiler/xla/service/gpu/tests:tree_reduction_rewriter_test PASSED in 2.0s //tensorflow/compiler/xla/service/graphcycles:graphcycles_test PASSED in 1.2s //tensorflow/compiler/xla/service/graphcycles:ordered_set_test PASSED in 0.2s //tensorflow/compiler/xla/service/llvm_ir:alias_analysis_test PASSED in 7.3s //tensorflow/compiler/xla/service/llvm_ir:ir_array_test PASSED in 0.5s //tensorflow/compiler/xla/service/spmd:canonicalize_all_gather_for_cse_test PASSED in 0.9s //tensorflow/compiler/xla/service/spmd:collective_permute_motion_test PASSED in 0.8s //tensorflow/compiler/xla/service/spmd:partition_assignment_test PASSED in 1.8s //tensorflow/compiler/xla/service/spmd:schedule_aware_collective_ops_cse_test PASSED in 1.2s //tensorflow/compiler/xla/service/spmd:spmd_partitioner_test PASSED in 4.5s //tensorflow/compiler/xla/service/spmd:spmd_prepare_test PASSED in 0.9s //tensorflow/compiler/xla/service/spmd:stateful_rng_spmd_partitioner_test PASSED in 0.8s //tensorflow/compiler/xla/stream_executor:dnn_test PASSED in 0.3s //tensorflow/compiler/xla/stream_executor:stream_test PASSED in 0.2s //tensorflow/compiler/xla/stream_executor/host:host_stream_test PASSED in 0.1s //tensorflow/compiler/xla/stream_executor/tpu:c_api_conversions_test PASSED in 0.3s //tensorflow/compiler/xla/tests:all_reduce_test_cpu PASSED in 6.8s //tensorflow/compiler/xla/tests:axpy_simple_test_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests:bad_rng_shape_validation_test_cpu PASSED in 6.6s //tensorflow/compiler/xla/tests:binop_scaling_test_cpu PASSED in 8.5s //tensorflow/compiler/xla/tests:bitcast_convert_test_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests:broadcast_simple_test_cpu PASSED in 10.7s //tensorflow/compiler/xla/tests:broadcast_test_cpu PASSED in 8.3s //tensorflow/compiler/xla/tests:buffer_donation_test_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests:call_test_cpu PASSED in 8.1s //tensorflow/compiler/xla/tests:check_execution_arity_test_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests:cholesky_test_cpu PASSED in 16.4s //tensorflow/compiler/xla/tests:client_test_cpu PASSED in 6.8s //tensorflow/compiler/xla/tests:collective_ops_test_cpu PASSED in 21.4s //tensorflow/compiler/xla/tests:collective_pipeliner_execution_test_cpu PASSED in 18.0s //tensorflow/compiler/xla/tests:compilation_cache_test_cpu PASSED in 5.7s //tensorflow/compiler/xla/tests:compute_constant_test_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests:concat_test_cpu PASSED in 8.7s //tensorflow/compiler/xla/tests:constant_reduction_function_test_cpu PASSED in 6.7s //tensorflow/compiler/xla/tests:constants_test_cpu PASSED in 8.2s //tensorflow/compiler/xla/tests:convert_test_cpu PASSED in 13.6s //tensorflow/compiler/xla/tests:copy_test_cpu PASSED in 8.1s //tensorflow/compiler/xla/tests:cpu_gpu_fusion_test_cpu PASSED in 11.9s //tensorflow/compiler/xla/tests:custom_call_test_cpu PASSED in 9.2s //tensorflow/compiler/xla/tests:deallocation_test_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests:deconstruct_tuple_test_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests:deep_graph_test_cpu PASSED in 10.3s //tensorflow/compiler/xla/tests:fft_test_cpu PASSED in 7.2s //tensorflow/compiler/xla/tests:float8_test_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests:floor_ceil_test_cpu PASSED in 6.5s //tensorflow/compiler/xla/tests:fmax_fmin_test_cpu PASSED in 8.5s //tensorflow/compiler/xla/tests:gather_operation_test_cpu PASSED in 15.3s //tensorflow/compiler/xla/tests:get_dimension_size_test_cpu PASSED in 9.2s //tensorflow/compiler/xla/tests:half_test_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests:hlo_metadata_test PASSED in 6.1s //tensorflow/compiler/xla/tests:literal_test_util_test PASSED in 5.6s //tensorflow/compiler/xla/tests:local_client_allocation_test_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests:local_client_aot_test PASSED in 0.1s //tensorflow/compiler/xla/tests:log_test_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests:map_test_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests:matrix_ops_simple_test_cpu PASSED in 24.4s //tensorflow/compiler/xla/tests:multidimensional_slice_test_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests:multiple_devices_on_host_test PASSED in 7.3s //tensorflow/compiler/xla/tests:multithreaded_compilation_test_cpu PASSED in 6.6s //tensorflow/compiler/xla/tests:outfeed_in_nested_computation_test_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests:pad_test_cpu PASSED in 8.8s //tensorflow/compiler/xla/tests:pred_test_cpu PASSED in 8.2s //tensorflow/compiler/xla/tests:query_inferred_shape_test_cpu PASSED in 6.8s //tensorflow/compiler/xla/tests:reduce_hlo_test_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests:reduce_precision_test_cpu PASSED in 13.5s //tensorflow/compiler/xla/tests:replay_test_cpu PASSED in 8.0s //tensorflow/compiler/xla/tests:reshape_motion_test_cpu PASSED in 8.3s //tensorflow/compiler/xla/tests:reverse_test_cpu PASSED in 8.6s //tensorflow/compiler/xla/tests:round_trip_packed_literal_test_cpu PASSED in 9.0s //tensorflow/compiler/xla/tests:round_trip_transfer_test_cpu PASSED in 10.5s //tensorflow/compiler/xla/tests:sample_text_test_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests:scatter_test_cpu PASSED in 13.2s //tensorflow/compiler/xla/tests:select_test_cpu PASSED in 7.2s //tensorflow/compiler/xla/tests:test_utils_test_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests:tile_assignment_test PASSED in 0.2s //tensorflow/compiler/xla/tests:token_hlo_test_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests:topk_test_cpu PASSED in 8.9s //tensorflow/compiler/xla/tests:transfer_manager_test_cpu PASSED in 14.6s //tensorflow/compiler/xla/tests:transpose_test_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests:tuple_test_cpu PASSED in 6.4s //tensorflow/compiler/xla/tests:unary_op_test_cpu PASSED in 7.0s //tensorflow/compiler/xla/tests:value_inference_test_cpu PASSED in 5.9s //tensorflow/compiler/xla/tests:vector_ops_reduce_test_cpu PASSED in 8.5s //tensorflow/compiler/xla/tests:vector_ops_simple_test_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests:while_test_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests/fuzz:rand_000000_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests/fuzz:rand_000003_cpu PASSED in 9.6s //tensorflow/compiler/xla/tests/fuzz:rand_000005_cpu PASSED in 8.6s //tensorflow/compiler/xla/tests/fuzz:rand_000006_cpu PASSED in 7.9s //tensorflow/compiler/xla/tests/fuzz:rand_000007_cpu PASSED in 7.2s //tensorflow/compiler/xla/tests/fuzz:rand_000008_cpu PASSED in 6.5s //tensorflow/compiler/xla/tests/fuzz:rand_000009_cpu PASSED in 8.0s //tensorflow/compiler/xla/tests/fuzz:rand_000013_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests/fuzz:rand_000015_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests/fuzz:rand_000016_cpu PASSED in 8.0s //tensorflow/compiler/xla/tests/fuzz:rand_000017_cpu PASSED in 8.4s //tensorflow/compiler/xla/tests/fuzz:rand_000018_cpu PASSED in 8.7s //tensorflow/compiler/xla/tests/fuzz:rand_000019_cpu PASSED in 20.8s //tensorflow/compiler/xla/tests/fuzz:rand_000020_cpu PASSED in 8.4s //tensorflow/compiler/xla/tests/fuzz:rand_000022_cpu PASSED in 8.1s //tensorflow/compiler/xla/tests/fuzz:rand_000024_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests/fuzz:rand_000025_cpu PASSED in 7.9s //tensorflow/compiler/xla/tests/fuzz:rand_000026_cpu PASSED in 6.7s //tensorflow/compiler/xla/tests/fuzz:rand_000030_cpu PASSED in 7.9s //tensorflow/compiler/xla/tests/fuzz:rand_000031_cpu PASSED in 8.1s //tensorflow/compiler/xla/tests/fuzz:rand_000032_cpu PASSED in 7.5s //tensorflow/compiler/xla/tests/fuzz:rand_000033_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests/fuzz:rand_000034_cpu PASSED in 8.8s //tensorflow/compiler/xla/tests/fuzz:rand_000035_cpu PASSED in 7.5s //tensorflow/compiler/xla/tests/fuzz:rand_000036_cpu PASSED in 8.2s //tensorflow/compiler/xla/tests/fuzz:rand_000039_cpu PASSED in 12.6s //tensorflow/compiler/xla/tests/fuzz:rand_000040_cpu PASSED in 6.8s //tensorflow/compiler/xla/tests/fuzz:rand_000041_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests/fuzz:rand_000043_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests/fuzz:rand_000049_cpu PASSED in 24.7s //tensorflow/compiler/xla/tests/fuzz:rand_000053_cpu PASSED in 6.1s //tensorflow/compiler/xla/tests/fuzz:rand_000056_cpu PASSED in 8.4s //tensorflow/compiler/xla/tests/fuzz:rand_000059_cpu PASSED in 7.5s //tensorflow/compiler/xla/tests/fuzz:rand_000061_cpu PASSED in 7.5s //tensorflow/compiler/xla/tests/fuzz:rand_000062_cpu PASSED in 8.6s //tensorflow/compiler/xla/tests/fuzz:rand_000064_cpu PASSED in 8.5s //tensorflow/compiler/xla/tests/fuzz:rand_000066_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests/fuzz:rand_000069_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests/fuzz:rand_000071_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests/fuzz:rand_000077_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests/fuzz:rand_000078_cpu PASSED in 10.4s //tensorflow/compiler/xla/tests/fuzz:rand_000079_cpu PASSED in 8.3s //tensorflow/compiler/xla/tests/fuzz:rand_000081_cpu PASSED in 6.0s //tensorflow/compiler/xla/tests/fuzz:rand_000084_cpu PASSED in 6.3s //tensorflow/compiler/xla/tests/fuzz:rand_000085_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests/fuzz:rand_000086_cpu PASSED in 6.7s //tensorflow/compiler/xla/tests/fuzz:rand_000088_cpu PASSED in 5.9s //tensorflow/compiler/xla/tests/fuzz:rand_000089_cpu PASSED in 7.0s //tensorflow/compiler/xla/tests/fuzz:rand_000090_cpu PASSED in 6.3s //tensorflow/compiler/xla/tests/fuzz:rand_000092_cpu PASSED in 8.1s //tensorflow/compiler/xla/tests/fuzz:rand_000094_cpu PASSED in 7.2s //tensorflow/compiler/xla/tests/fuzz:rand_000095_cpu PASSED in 8.2s //tensorflow/compiler/xla/tools:hlo_control_flow_flattening_test PASSED in 0.7s //tensorflow/compiler/xla/tools:hlo_extractor_test PASSED in 0.9s //tensorflow/compiler/xla/tools:hlo_module_loader_test PASSED in 1.2s //tensorflow/compiler/xla/tools:hlo_slicer_test PASSED in 0.6s //tensorflow/compiler/xla/tools:interactive_graphviz_bin_test PASSED in 0.2s //tensorflow/compiler/xla/tools:run_hlo_module_bin_test PASSED in 0.3s //tensorflow/compiler/xla/tools/hlo_bisect:hlo_bisect_state_test PASSED in 0.8s //tensorflow/compiler/xla/translate/hlo_to_mhlo:hlo_utils_test PASSED in 0.9s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:bool_compare.hlotxt.test PASSED in 2.0s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:case_conditional.hlotxt.test PASSED in 0.8s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:dynamic_param.hlo.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:entry_computation_layout.hlotxt.test PASSED in 1.1s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:frontend_attributes.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fully_connected_reference_model.hlotxt.test PASSED in 0.8s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fusion.hlotxt.test PASSED in 1.2s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:if_conditional.hlotxt.test PASSED in 0.6s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import.hlotxt.test PASSED in 3.9s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import_async.hlotxt.test PASSED in 0.7s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:layouts_and_names.hlotxt.test PASSED in 0.6s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:location.hlotxt.test PASSED in 0.7s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:module_attributes.hlo.test PASSED in 0.4s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:send_recv.hlotxt.test PASSED in 1.4s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:simple.hlo.test PASSED in 1.6s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:spmd_module_sharding.hlo.test PASSED in 0.8s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:stacktrace_to_location.hlo.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:types.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:while.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo:type_to_shape_test PASSED in 0.9s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:add.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:case.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:dynamic.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export-with-layouts.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export.mlir.test PASSED in 2.8s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_and_check_layouts.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_large_constants.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_replicas.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:frontend_attributes.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:fusion.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:if.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:input_output_aliasing.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:layouts_and_names.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:location_to_op_metadata.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:location_to_stacktrace.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:missing_main.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:module_attributes.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:multiple_return_tuple.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:opaque_elements_attr.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:rng_get_and_update_state.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:sharding.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:simple.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:unsupported_type.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:while.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:hlo_text_to_lhlo_no_opt.hlotxt.test PASSED in 1.0s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:no_opt_ops.hlotxt.test PASSED in 0.9s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:non_identity_layouts.hlotxt.test PASSED in 0.4s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:ops.mlir.test PASSED in 1.8s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:passthrough.mlir.test PASSED in 0.5s //tensorflow/core:__tensorflow_core_lib_core_legacy_lib_core_all_tests PASSED in 6.2s //tensorflow/core:__tensorflow_core_lib_gtl_legacy_lib_gtl_tests PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_cell_reader_test PASSED in 35.1s //tensorflow/core:__tensorflow_core_lib_monitoring_collection_registry_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_monitoring_counter_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_gauge_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_metric_def_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_percentile_sampler_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_sampler_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_monitoring_test_utils_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_strings_legacy_low_level_library_tests PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_wav_wav_io_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_util_mkl_util_test_srcs PASSED in 0.1s //tensorflow/core:__tensorflow_tsl_lib_core_legacy_lib_core_all_tests PASSED in 0.5s //tensorflow/core:lib_strings_ordered_code_test PASSED in 1.8s //tensorflow/core:lib_strings_proto_serialization_test PASSED in 0.1s //tensorflow/core/api_def:api_test PASSED in 3.5s //tensorflow/core/api_def:update_api_def_test PASSED in 0.2s //tensorflow/core/common_runtime:all_to_all_test_cpu PASSED in 1.4s //tensorflow/core/common_runtime:arg_ret_placement_test PASSED in 0.5s //tensorflow/core/common_runtime:buf_rendezvous_test PASSED in 0.8s //tensorflow/core/common_runtime:collective_executor_mgr_test PASSED in 0.9s //tensorflow/core/common_runtime:collective_param_resolver_local_test PASSED in 5.0s //tensorflow/core/common_runtime:collective_rma_local_test PASSED in 1.3s //tensorflow/core/common_runtime:composite_device_test PASSED in 0.5s //tensorflow/core/common_runtime:cost_measurement_registry_test PASSED in 2.1s //tensorflow/core/common_runtime:cost_util_test PASSED in 0.1s //tensorflow/core/common_runtime:device_mgr_test PASSED in 1.0s //tensorflow/core/common_runtime:device_propagation_test PASSED in 0.6s //tensorflow/core/common_runtime:device_resolver_local_test PASSED in 1.2s //tensorflow/core/common_runtime:device_set_test PASSED in 0.9s //tensorflow/core/common_runtime:direct_session_test_cpu PASSED in 1.8s //tensorflow/core/common_runtime:direct_session_with_debug_test PASSED in 1.9s //tensorflow/core/common_runtime:direct_session_with_tracking_alloc_test PASSED in 1.0s //tensorflow/core/common_runtime:dynamic_device_mgr_test PASSED in 0.9s //tensorflow/core/common_runtime:eval_const_tensor_test PASSED in 0.5s //tensorflow/core/common_runtime:executor_test PASSED in 2.6s //tensorflow/core/common_runtime:function_optimization_registration_test PASSED in 1.0s //tensorflow/core/common_runtime:function_optimization_registry_no_pass_test PASSED in 1.0s //tensorflow/core/common_runtime:function_optimization_registry_pass_failure_test PASSED in 0.8s //tensorflow/core/common_runtime:function_optimization_registry_test PASSED in 2.4s //tensorflow/core/common_runtime:function_threadpool_test PASSED in 0.9s //tensorflow/core/common_runtime:graph_constructor_test PASSED in 2.7s //tensorflow/core/common_runtime:graph_runner_test PASSED in 1.0s //tensorflow/core/common_runtime:hierarchical_tree_broadcaster_test_cpu PASSED in 3.0s //tensorflow/core/common_runtime:inline_function_utils_test PASSED in 0.6s //tensorflow/core/common_runtime:input_colocation_exemption_registry_test PASSED in 0.5s //tensorflow/core/common_runtime:int32_fulltype_test PASSED in 1.7s //tensorflow/core/common_runtime:isolate_placer_inspection_required_ops_pass_test PASSED in 1.0s //tensorflow/core/common_runtime:lower_case_op_test PASSED in 1.7s //tensorflow/core/common_runtime:lower_function_call_test PASSED in 1.9s //tensorflow/core/common_runtime:lower_functional_ops_test PASSED in 1.6s //tensorflow/core/common_runtime:lower_if_op_test PASSED in 3.1s //tensorflow/core/common_runtime:lower_while_op_test PASSED in 2.7s //tensorflow/core/common_runtime:mkl_cpu_allocator_test PASSED in 0.3s //tensorflow/core/common_runtime:mkl_threadpool_device_test PASSED in 0.1s //tensorflow/core/common_runtime:no_op_cost_measurement_test PASSED in 0.1s //tensorflow/core/common_runtime:null_request_cost_accessor_test PASSED in 0.1s //tensorflow/core/common_runtime:optimization_registry_test PASSED in 2.2s //tensorflow/core/common_runtime:optimize_cross_host_control_deps_test PASSED in 6.7s //tensorflow/core/common_runtime:optimize_function_graph_utils_test PASSED in 0.6s //tensorflow/core/common_runtime:partitioning_utils_test PASSED in 0.9s //tensorflow/core/common_runtime:pending_counts_test PASSED in 0.9s //tensorflow/core/common_runtime:permuter_test_cpu PASSED in 4.9s //tensorflow/core/common_runtime:placer_inspection_required_ops_utils_test PASSED in 1.1s //tensorflow/core/common_runtime:placer_test PASSED in 0.7s //tensorflow/core/common_runtime:process_function_library_runtime_test_cpu PASSED in 0.8s //tensorflow/core/common_runtime:process_util_test PASSED in 0.3s //tensorflow/core/common_runtime:quantize_training_test PASSED in 1.9s //tensorflow/core/common_runtime:rendezvous_util_test PASSED in 0.1s //tensorflow/core/common_runtime:replicate_per_replica_nodes_test PASSED in 0.5s //tensorflow/core/common_runtime:request_cost_accessor_registry_test PASSED in 2.2s //tensorflow/core/common_runtime:request_cost_test PASSED in 0.1s //tensorflow/core/common_runtime:ring_gatherer_test_cpu PASSED in 4.1s //tensorflow/core/common_runtime:ring_reducer_test_cpu PASSED in 6.4s //tensorflow/core/common_runtime:scoped_allocator_mgr_test PASSED in 4.4s //tensorflow/core/common_runtime:session_test PASSED in 1.4s //tensorflow/core/common_runtime:shape_refiner_test PASSED in 1.4s //tensorflow/core/common_runtime:single_threaded_executor_test PASSED in 1.3s //tensorflow/core/common_runtime:threadpool_device_test PASSED in 0.9s //tensorflow/core/common_runtime:type_inference_test PASSED in 2.6s //tensorflow/core/common_runtime/eager:attr_builder_test PASSED in 26.8s //tensorflow/core/common_runtime/eager:context_test PASSED in 11.4s //tensorflow/core/common_runtime/eager:custom_device_test PASSED in 11.5s //tensorflow/core/common_runtime/eager:eager_executor_test PASSED in 13.1s //tensorflow/core/common_runtime/eager:eager_op_rewrite_registry_test PASSED in 0.7s //tensorflow/core/common_runtime/eager:eager_operation_test PASSED in 13.2s //tensorflow/core/common_runtime/eager:execute_node_test PASSED in 16.7s //tensorflow/core/common_runtime/eager:execute_test PASSED in 23.3s //tensorflow/core/common_runtime/eager:kernel_and_device_test PASSED in 1.6s //tensorflow/core/common_runtime/eager:mkl_eager_op_rewrite_test PASSED in 13.2s //tensorflow/core/common_runtime/eager:placement_test PASSED in 11.2s //tensorflow/core/common_runtime/eager:placement_utils_test PASSED in 12.2s //tensorflow/core/common_runtime/eager:summary_optimizer_test PASSED in 0.1s //tensorflow/core/common_runtime/eager:tensor_handle_data_test PASSED in 11.9s //tensorflow/core/common_runtime/eager:tensor_handle_test PASSED in 10.8s //tensorflow/core/common_runtime/gpu:gpu_device_on_non_gpu_machine_test PASSED in 0.1s //tensorflow/core/common_runtime/next_pluggable_device/c:plugin_c_api_test PASSED in 24.4s //tensorflow/core/config:flags_py_test PASSED in 27.3s //tensorflow/core/config:flags_test PASSED in 0.3s //tensorflow/core/data:compression_utils_test PASSED in 1.8s //tensorflow/core/data:dataset_utils_test PASSED in 0.5s //tensorflow/core/data:hash_utils_test PASSED in 1.0s //tensorflow/core/data:metric_utils_test PASSED in 5.7s //tensorflow/core/data:name_utils_test PASSED in 0.1s //tensorflow/core/data:rewrite_utils_test PASSED in 0.6s //tensorflow/core/data:serialization_utils_test PASSED in 0.7s //tensorflow/core/data:snapshot_utils_test PASSED in 8.5s //tensorflow/core/data:split_utils_test PASSED in 0.6s //tensorflow/core/data:standalone_save_restore_test PASSED in 3.5s //tensorflow/core/data:standalone_test PASSED in 3.1s //tensorflow/core/data:tfdataz_metrics_test PASSED in 3.1s //tensorflow/core/data:unbounded_thread_pool_test PASSED in 0.4s //tensorflow/core/data/service:auto_scaler_test PASSED in 0.2s //tensorflow/core/data/service:auto_shard_rewriter_test PASSED in 0.7s //tensorflow/core/data/service:common_test PASSED in 0.3s //tensorflow/core/data/service:credentials_factory_test PASSED in 1.0s //tensorflow/core/data/service:cross_trainer_cache_test PASSED in 1.4s //tensorflow/core/data/service:data_service_test PASSED in 13.7s //tensorflow/core/data/service:data_transfer_test PASSED in 0.7s //tensorflow/core/data/service:dataset_store_test PASSED in 0.6s //tensorflow/core/data/service:dispatcher_client_test PASSED in 3.9s //tensorflow/core/data/service:dispatcher_state_test PASSED in 0.5s //tensorflow/core/data/service:grpc_dispatcher_impl_test PASSED in 2.1s //tensorflow/core/data/service:grpc_util_test PASSED in 1.1s //tensorflow/core/data/service:grpc_worker_impl_test PASSED in 2.5s //tensorflow/core/data/service:journal_test PASSED in 0.8s //tensorflow/core/data/service:logging_utils_test PASSED in 0.1s //tensorflow/core/data/service:task_runner_test PASSED in 2.9s //tensorflow/core/data/service:test_util_test PASSED in 2.2s //tensorflow/core/data/service:url_test PASSED in 0.1s //tensorflow/core/data/service:utils_test PASSED in 0.6s //tensorflow/core/data/service:validate_utils_test PASSED in 0.1s //tensorflow/core/data/service:worker_client_test PASSED in 3.6s //tensorflow/core/data/service:worker_impl_test PASSED in 3.1s //tensorflow/core/data/service/client:data_service_client_test PASSED in 5.4s //tensorflow/core/data/service/client:utils_test PASSED in 3.2s //tensorflow/core/data/service/client:validate_utils_test PASSED in 1.2s //tensorflow/core/data/service/snapshot:distributed_snapshot_test PASSED in 18.5s //tensorflow/core/data/service/snapshot:file_utils_test PASSED in 0.6s //tensorflow/core/data/service/snapshot:path_utils_test PASSED in 0.2s //tensorflow/core/data/service/snapshot:snapshot_manager_test PASSED in 2.1s //tensorflow/core/data/service/snapshot:snapshot_split_provider_test PASSED in 0.9s //tensorflow/core/data/service/snapshot:snapshot_stream_writer_checkpoint_test PASSED in 3.9s //tensorflow/core/data/service/snapshot:snapshot_stream_writer_test PASSED in 3.0s //tensorflow/core/data/service/snapshot:utils_test PASSED in 0.2s //tensorflow/core/debug:debug_graph_utils_test PASSED in 0.5s //tensorflow/core/distributed_runtime:call_options_test PASSED in 0.7s //tensorflow/core/distributed_runtime:cluster_function_library_runtime_test PASSED in 21.3s //tensorflow/core/distributed_runtime:collective_param_resolver_distributed_test PASSED in 1.0s //tensorflow/core/distributed_runtime:collective_rma_distributed_test PASSED in 0.9s //tensorflow/core/distributed_runtime:device_resolver_distributed_test PASSED in 0.5s //tensorflow/core/distributed_runtime:message_wrappers_test PASSED in 0.1s //tensorflow/core/distributed_runtime:partial_run_mgr_test PASSED in 0.4s //tensorflow/core/distributed_runtime:recent_request_ids_test PASSED in 0.1s //tensorflow/core/distributed_runtime:request_id_test PASSED in 0.1s //tensorflow/core/distributed_runtime:rpc_collective_executor_mgr_test PASSED in 0.6s //tensorflow/core/distributed_runtime:server_lib_test PASSED in 0.1s //tensorflow/core/distributed_runtime:session_mgr_test PASSED in 0.9s //tensorflow/core/distributed_runtime:tensor_coding_test PASSED in 0.1s //tensorflow/core/distributed_runtime/coordination:coordination_service_barrier_proxy_test PASSED in 2.3s //tensorflow/core/distributed_runtime/eager:eager_service_impl_test PASSED in 31.5s //tensorflow/core/distributed_runtime/eager:remote_mgr_test PASSED in 10.2s //tensorflow/core/distributed_runtime/integration_test:c_api_multi_client_test_cpu PASSED in 31.8s //tensorflow/core/distributed_runtime/integration_test:c_api_recoverable_jobs_test_cpu PASSED in 41.5s //tensorflow/core/distributed_runtime/integration_test:c_api_session_coordination_test_cpu PASSED in 31.3s //tensorflow/core/distributed_runtime/rpc:grpc_tensor_coding_test PASSED in 2.7s //tensorflow/core/distributed_runtime/rpc:grpc_worker_cache_test PASSED in 0.5s //tensorflow/core/distributed_runtime/rpc/eager:grpc_eager_client_test PASSED in 2.0s //tensorflow/core/example:example_parser_configuration_test PASSED in 1.8s //tensorflow/core/example:feature_util_test PASSED in 0.1s //tensorflow/core/framework:allocator_test PASSED in 4.8s //tensorflow/core/framework:attr_value_util_test PASSED in 1.5s //tensorflow/core/framework:batch_util_test PASSED in 0.7s //tensorflow/core/framework:bfloat16_test PASSED in 0.8s //tensorflow/core/framework:common_shape_fns_test PASSED in 0.9s //tensorflow/core/framework:dataset_test PASSED in 1.7s //tensorflow/core/framework:device_base_test PASSED in 1.0s //tensorflow/core/framework:disable_jit_test PASSED in 0.9s //tensorflow/core/framework:framework_op_gen_lib_test PASSED in 0.2s //tensorflow/core/framework:framework_op_segment_test PASSED in 0.9s //tensorflow/core/framework:framework_resource_var_test PASSED in 0.2s //tensorflow/core/framework:framework_run_handler_test PASSED in 1.1s //tensorflow/core/framework:framework_run_handler_util_test PASSED in 2.1s //tensorflow/core/framework:full_type_inference_util_test PASSED in 1.3s //tensorflow/core/framework:full_type_util_test PASSED in 1.0s //tensorflow/core/framework:function_test PASSED in 0.9s //tensorflow/core/framework:graph_def_util_test PASSED in 0.9s //tensorflow/core/framework:graph_to_functiondef_test PASSED in 1.1s //tensorflow/core/framework:kernel_def_builder_test PASSED in 0.6s //tensorflow/core/framework:kernel_def_util_test PASSED in 0.9s //tensorflow/core/framework:memory_types_test PASSED in 0.7s //tensorflow/core/framework:model_test PASSED in 1.6s //tensorflow/core/framework:node_def_builder_test PASSED in 1.0s //tensorflow/core/framework:node_def_util_test PASSED in 0.7s //tensorflow/core/framework:node_properties_test PASSED in 1.3s //tensorflow/core/framework:op_compatibility_test PASSED in 0.8s //tensorflow/core/framework:op_def_builder_test PASSED in 0.7s //tensorflow/core/framework:op_def_util_test PASSED in 0.9s //tensorflow/core/framework:op_kernel_test PASSED in 1.4s //tensorflow/core/framework:op_registration_test PASSED in 0.9s //tensorflow/core/framework:partial_tensor_shape_test PASSED in 0.8s //tensorflow/core/framework:rendezvous_test PASSED in 4.3s //tensorflow/core/framework:resource_handle_test PASSED in 0.2s //tensorflow/core/framework:resource_mgr_test PASSED in 2.1s //tensorflow/core/framework:resource_op_kernel_test PASSED in 1.1s //tensorflow/core/framework:shape_inference_test PASSED in 1.3s //tensorflow/core/framework:shape_inference_testutil_test PASSED in 0.8s //tensorflow/core/framework:tensor_shape_test PASSED in 6.7s //tensorflow/core/framework:tensor_slice_test PASSED in 0.7s //tensorflow/core/framework:tensor_test PASSED in 29.2s //tensorflow/core/framework:tensor_testutil_test PASSED in 0.6s //tensorflow/core/framework:tensor_util_test PASSED in 1.1s //tensorflow/core/framework:tracking_allocator_test PASSED in 0.8s //tensorflow/core/framework:types_test PASSED in 1.0s //tensorflow/core/framework:variant_op_registry_test PASSED in 23.8s //tensorflow/core/framework:variant_test PASSED in 0.9s //tensorflow/core/framework/registration:registration_test PASSED in 0.7s //tensorflow/core/function/capture:by_ref_capture_test PASSED in 14.7s //tensorflow/core/function/capture:capture_container_test PASSED in 9.4s //tensorflow/core/function/integration_test:side_inputs_manual_api_test PASSED in 24.2s //tensorflow/core/function/integration_test:side_inputs_test PASSED in 17.4s //tensorflow/core/function/polymorphism:function_cache_test PASSED in 14.4s //tensorflow/core/function/polymorphism:function_type_test PASSED in 16.0s //tensorflow/core/function/polymorphism:type_dispatch_test PASSED in 14.4s //tensorflow/core/function/runtime_client:runtime_client_cc_test PASSED in 41.1s //tensorflow/core/function/trace_type:default_types_test PASSED in 25.2s //tensorflow/core/function/trace_type:serialization_test PASSED in 10.4s //tensorflow/core/function/trace_type:trace_type_test PASSED in 22.1s //tensorflow/core/graph:algorithm_test PASSED in 0.9s //tensorflow/core/graph:collective_order_test PASSED in 1.6s //tensorflow/core/graph:control_flow_test PASSED in 1.1s //tensorflow/core/graph:costmodel_test PASSED in 0.8s //tensorflow/core/graph:edgeset_test PASSED in 1.2s //tensorflow/core/graph:graph_debug_info_builder_test PASSED in 1.8s //tensorflow/core/graph:graph_def_builder_test PASSED in 1.3s //tensorflow/core/graph:graph_partition_test PASSED in 1.0s //tensorflow/core/graph:graph_test PASSED in 1.2s //tensorflow/core/graph:node_builder_test PASSED in 0.8s //tensorflow/core/graph:optimizer_cse_test PASSED in 0.8s //tensorflow/core/graph:subgraph_test PASSED in 0.8s //tensorflow/core/graph:tensor_id_test PASSED in 0.8s //tensorflow/core/graph:validate_test PASSED in 1.2s //tensorflow/core/graph/regularization:simple_delete_test PASSED in 0.4s //tensorflow/core/graph/regularization:util_test PASSED in 0.6s //tensorflow/core/grappler:graph_topology_view_test PASSED in 1.1s //tensorflow/core/grappler:graph_view_test PASSED in 1.4s //tensorflow/core/grappler:grappler_item_builder_test PASSED in 1.5s //tensorflow/core/grappler:grappler_item_test PASSED in 1.2s //tensorflow/core/grappler:mutable_graph_view_test PASSED in 1.6s //tensorflow/core/grappler:utils_test PASSED in 3.6s //tensorflow/core/grappler/clusters:single_machine_test PASSED in 23.7s //tensorflow/core/grappler/clusters:virtual_cluster_test PASSED in 1.5s //tensorflow/core/grappler/costs:analytical_cost_estimator_test PASSED in 1.9s //tensorflow/core/grappler/costs:cost_estimator_test PASSED in 0.2s //tensorflow/core/grappler/costs:graph_memory_test PASSED in 1.7s //tensorflow/core/grappler/costs:graph_properties_test PASSED in 4.5s //tensorflow/core/grappler/costs:robust_stats_test PASSED in 0.1s //tensorflow/core/grappler/costs:utils_test PASSED in 1.3s //tensorflow/core/grappler/costs:virtual_placer_test PASSED in 0.4s //tensorflow/core/grappler/costs:virtual_scheduler_test PASSED in 1.8s //tensorflow/core/grappler/graph_analyzer:gen_node_test PASSED in 2.2s //tensorflow/core/grappler/graph_analyzer:graph_analyzer_test PASSED in 2.0s //tensorflow/core/grappler/graph_analyzer:hash_tools_test PASSED in 1.5s //tensorflow/core/grappler/graph_analyzer:sig_node_test PASSED in 2.9s //tensorflow/core/grappler/graph_analyzer:subgraph_test PASSED in 1.6s //tensorflow/core/grappler/inputs:utils_test PASSED in 0.1s //tensorflow/core/grappler/optimizers:arithmetic_optimizer_test_cpu PASSED in 3.5s //tensorflow/core/grappler/optimizers:auto_mixed_precision_test_cpu PASSED in 1.6s //tensorflow/core/grappler/optimizers:auto_parallel_test_cpu PASSED in 1.6s //tensorflow/core/grappler/optimizers:common_subgraph_elimination_test_cpu PASSED in 1.7s //tensorflow/core/grappler/optimizers:custom_graph_optimizer_registry_test_cpu PASSED in 4.7s //tensorflow/core/grappler/optimizers:debug_stripper_test_cpu PASSED in 2.1s //tensorflow/core/grappler/optimizers:dependency_optimizer_test_cpu PASSED in 1.4s //tensorflow/core/grappler/optimizers:evaluation_utils_test PASSED in 0.5s //tensorflow/core/grappler/optimizers:function_api_info_test PASSED in 0.1s //tensorflow/core/grappler/optimizers:function_optimizer_test_cpu PASSED in 2.6s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_test_cpu PASSED in 2.0s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_factory_test PASSED in 1.5s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_test_cpu PASSED in 1.9s //tensorflow/core/grappler/optimizers:graph_optimizer_stage_test_cpu PASSED in 1.6s //tensorflow/core/grappler/optimizers:implementation_selector_test PASSED in 1.7s //tensorflow/core/grappler/optimizers:loop_optimizer_test_cpu PASSED in 1.4s //tensorflow/core/grappler/optimizers:memory_optimizer_test_cpu PASSED in 1.6s //tensorflow/core/grappler/optimizers:meta_optimizer_test_cpu PASSED in 7.4s //tensorflow/core/grappler/optimizers:mkl_remapper_test PASSED in 4.1s //tensorflow/core/grappler/optimizers:model_pruner_test_cpu PASSED in 2.9s //tensorflow/core/grappler/optimizers:pin_to_host_optimizer_test_cpu PASSED in 1.7s //tensorflow/core/grappler/optimizers:remapper_test_cpu PASSED in 2.7s //tensorflow/core/grappler/optimizers:scoped_allocator_optimizer_test PASSED in 1.6s //tensorflow/core/grappler/optimizers:shape_optimizer_test_cpu PASSED in 2.2s //tensorflow/core/grappler/optimizers:static_schedule_test_cpu PASSED in 2.4s //tensorflow/core/grappler/optimizers:tfg_optimizer_hook_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:auto_shard_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:autotune_buffer_sizes_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:batch_parallelization_test PASSED in 1.2s //tensorflow/core/grappler/optimizers/data:disable_intra_op_parallelism_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:disable_prefetch_legacy_autotune_test PASSED in 0.9s //tensorflow/core/grappler/optimizers/data:enable_gradient_descent_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:filter_fusion_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:filter_parallelization_test PASSED in 1.5s //tensorflow/core/grappler/optimizers/data:function_utils_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:fusion_utils_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:graph_utils_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:inject_prefetch_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:make_deterministic_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:make_sloppy_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:map_and_batch_fusion_test PASSED in 0.4s //tensorflow/core/grappler/optimizers/data:map_and_filter_fusion_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:map_fusion_test PASSED in 1.0s //tensorflow/core/grappler/optimizers/data:map_parallelization_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:noop_elimination_test PASSED in 1.0s //tensorflow/core/grappler/optimizers/data:parallel_batch_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:replicate_on_split_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:shuffle_and_repeat_fusion_test PASSED in 0.4s //tensorflow/core/grappler/optimizers/data:slack_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:split_utils_test PASSED in 1.0s //tensorflow/core/grappler/optimizers/data:use_private_thread_pool_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/inference:batch_op_rewriter_test PASSED in 0.2s //tensorflow/core/grappler/utils:canonicalizer_test PASSED in 1.5s //tensorflow/core/grappler/utils:colocation_test PASSED in 0.5s //tensorflow/core/grappler/utils:frame_test PASSED in 0.2s //tensorflow/core/grappler/utils:functions_test PASSED in 1.3s //tensorflow/core/grappler/utils:graph_view_internal_test PASSED in 0.5s //tensorflow/core/grappler/utils:graph_view_test PASSED in 1.2s //tensorflow/core/grappler/utils:grappler_test_test PASSED in 6.0s //tensorflow/core/grappler/utils:pattern_utils_test PASSED in 0.6s //tensorflow/core/grappler/utils:scc_test PASSED in 1.9s //tensorflow/core/grappler/utils:symbolic_shapes_test PASSED in 0.3s //tensorflow/core/grappler/utils:topological_sort_test PASSED in 0.5s //tensorflow/core/grappler/utils:tpu_test PASSED in 0.1s //tensorflow/core/grappler/utils:transitive_fanin_test PASSED in 0.6s //tensorflow/core/grappler/utils:traversal_test PASSED in 0.5s //tensorflow/core/grappler/verifiers:structure_verifier_test PASSED in 2.3s //tensorflow/core/ir:interfaces_test PASSED in 0.4s //tensorflow/core/ir:ops_test PASSED in 0.2s //tensorflow/core/ir:shape_inference_utils_test PASSED in 0.8s //tensorflow/core/ir:tf_op_registry_test PASSED in 0.3s //tensorflow/core/ir:tf_op_wrapper_test PASSED in 0.2s //tensorflow/core/ir:utility_test PASSED in 0.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:arg_as_control_ret.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:backedge_segment.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:empty.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:error_during_backedge.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_case_with_attr_inference.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_if_with_attr_inference.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_iterator_get_next_attr_inference.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_underscore_output_shapes.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_while_with_attr_inference.pbtxt.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infeed_dequeue.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_arg_handle_type.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_with_output_shapes.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_arg_name.pbtxt.test PASSED in 2.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_backedge_input_size.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_duplicated_node_name.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_index.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_name.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_attr_key.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_key.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_name.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_op_type.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_func_with_empty_name.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_function_import.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_control_result.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_input.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_name.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_result.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_attr_name.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_named_edge_index.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_handle_data.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_input.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result_value.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result_value.pbtxt.test PASSED in 2.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_input.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_two_inputs.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_named_edge_index.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_op_name.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_type_list.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:legacy_call.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_shape.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_zero_constant.pbtxt.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:three_nodes_with_attrs.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:version.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:empty.mlir.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:fulltype.mlir.test PASSED in 1.3s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:func_with_no_args_or_results.mlir.test PASSED in 1.1s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:negative_zero_constant.mlir.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:nested_legacy_call.mlir.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:three_nodes_with_attrs.mlir.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:version.mlir.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/saved_model:saved_model_roundtrip_test PASSED in 0.9s //tensorflow/core/ir/tests:attributes.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:canonicalize.mlir.test PASSED in 1.3s //tensorflow/core/ir/tests:compatible_types.mlir.test PASSED in 1.1s //tensorflow/core/ir/tests:concrete-ops.mlir.test PASSED in 1.3s //tensorflow/core/ir/tests:generic_concrete_ops.mlir.test PASSED in 1.3s //tensorflow/core/ir/tests:invalid-concrete-ops.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:invalid-preserved-attrs.mlir.test PASSED in 0.7s //tensorflow/core/ir/tests:invalid.mlir.test PASSED in 0.9s //tensorflow/core/ir/tests:invalid_types.mlir.test PASSED in 1.0s //tensorflow/core/ir/tests:ops.mlir.test PASSED in 1.6s //tensorflow/core/ir/tests:region-invalid-ops.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:region-ops-graph.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:region-ops.mlir.test PASSED in 0.8s //tensorflow/core/ir/tests:types.mlir.test PASSED in 0.4s //tensorflow/core/ir/types:dialect_test PASSED in 0.2s //tensorflow/core/kernels:as_string_op_test PASSED in 0.6s //tensorflow/core/kernels:basic_ops_benchmark_test PASSED in 0.5s //tensorflow/core/kernels:batch_kernels_env_test PASSED in 0.7s //tensorflow/core/kernels:batch_kernels_test PASSED in 32.6s //tensorflow/core/kernels:bias_op_test PASSED in 0.8s //tensorflow/core/kernels:bincount_op_test_cpu PASSED in 0.4s //tensorflow/core/kernels:broadcast_to_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:cast_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:checkpoint_callback_manager_test PASSED in 0.9s //tensorflow/core/kernels:clustering_ops_test PASSED in 0.6s //tensorflow/core/kernels:composite_tensor_variant_test PASSED in 0.9s //tensorflow/core/kernels:concat_op_test PASSED in 0.8s //tensorflow/core/kernels:constant_op_test_cpu PASSED in 1.3s //tensorflow/core/kernels:control_flow_ops_test PASSED in 6.4s //tensorflow/core/kernels:conv_grad_filter_ops_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels:conv_grad_input_ops_benchmark_test_cpu PASSED in 0.7s //tensorflow/core/kernels:conv_ops_benchmark_test_cpu PASSED in 0.6s //tensorflow/core/kernels:conv_ops_test_cpu PASSED in 8.5s //tensorflow/core/kernels:count_ops_test PASSED in 1.1s //tensorflow/core/kernels:cross_op_test PASSED in 0.8s //tensorflow/core/kernels:cwise_ops_test_cpu PASSED in 0.6s //tensorflow/core/kernels:debug_ops_test PASSED in 0.8s //tensorflow/core/kernels:decode_wav_op_test PASSED in 1.4s //tensorflow/core/kernels:deep_conv2d_test PASSED in 0.7s //tensorflow/core/kernels:dequantize_op_test PASSED in 2.6s //tensorflow/core/kernels:diag_op_test_cpu PASSED in 1.0s //tensorflow/core/kernels:dynamic_partition_op_test_cpu PASSED in 1.7s //tensorflow/core/kernels:dynamic_stitch_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:eigen_activations_test PASSED in 0.3s //tensorflow/core/kernels:eigen_attention_test PASSED in 0.2s //tensorflow/core/kernels:eigen_backward_cuboid_convolutions_test PASSED in 0.5s //tensorflow/core/kernels:eigen_backward_spatial_convolutions_test PASSED in 0.1s //tensorflow/core/kernels:eigen_benchmark_cpu_test PASSED in 0.7s //tensorflow/core/kernels:eigen_mkldnn_contraction_kernel_test PASSED in 0.1s //tensorflow/core/kernels:eigen_pooling_test PASSED in 0.4s //tensorflow/core/kernels:encode_wav_op_test PASSED in 1.8s //tensorflow/core/kernels:fingerprint_op_test PASSED in 1.5s //tensorflow/core/kernels:fused_batch_norm_ex_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:fused_batch_norm_op_test_cpu PASSED in 1.8s //tensorflow/core/kernels:gather_nd_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:gather_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:guarantee_const_op_test PASSED in 0.8s //tensorflow/core/kernels:identity_n_op_test PASSED in 0.8s //tensorflow/core/kernels:identity_op_test PASSED in 0.4s //tensorflow/core/kernels:immutable_constant_op_test PASSED in 1.2s //tensorflow/core/kernels:in_topk_op_test PASSED in 0.5s //tensorflow/core/kernels:isotonic_regression_op_test PASSED in 0.6s //tensorflow/core/kernels:logging_ops_test PASSED in 1.7s //tensorflow/core/kernels:lookup_ops_test PASSED in 0.6s //tensorflow/core/kernels:loss_test PASSED in 0.1s //tensorflow/core/kernels:lrn_op_test_cpu PASSED in 1.1s //tensorflow/core/kernels:matmul_op_test_cpu PASSED in 3.2s //tensorflow/core/kernels:merge_v2_checkpoints_op_test PASSED in 1.3s //tensorflow/core/kernels:mfcc_dct_test PASSED in 0.1s //tensorflow/core/kernels:mfcc_mel_filterbank_test PASSED in 0.1s //tensorflow/core/kernels:mfcc_op_test_cpu PASSED in 2.6s //tensorflow/core/kernels:mfcc_test PASSED in 0.1s //tensorflow/core/kernels:multinomial_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:nn_ops_test_cpu PASSED in 0.8s //tensorflow/core/kernels:one_hot_op_test PASSED in 1.2s //tensorflow/core/kernels:ops_testutil_test PASSED in 1.0s //tensorflow/core/kernels:ops_util_test PASSED in 0.1s //tensorflow/core/kernels:parameterized_truncated_normal_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:parse_tensor_test PASSED in 1.0s //tensorflow/core/kernels:quantization_utils_test PASSED in 0.7s //tensorflow/core/kernels:quantize_and_dequantize_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:quantize_down_and_shrink_range_op_test PASSED in 0.5s //tensorflow/core/kernels:quantize_op_test PASSED in 1.0s //tensorflow/core/kernels:quantized_activation_ops_test PASSED in 0.9s //tensorflow/core/kernels:quantized_add_op_test PASSED in 1.2s //tensorflow/core/kernels:quantized_batch_norm_op_test PASSED in 0.7s //tensorflow/core/kernels:quantized_bias_add_op_test PASSED in 0.5s //tensorflow/core/kernels:quantized_concat_op_test PASSED in 0.7s //tensorflow/core/kernels:quantized_conv_ops_test PASSED in 1.2s //tensorflow/core/kernels:quantized_instance_norm_test PASSED in 0.9s //tensorflow/core/kernels:quantized_matmul_op_test PASSED in 0.6s //tensorflow/core/kernels:quantized_mul_op_test PASSED in 1.3s //tensorflow/core/kernels:quantized_pooling_ops_test PASSED in 0.6s //tensorflow/core/kernels:quantized_reshape_op_test PASSED in 0.6s //tensorflow/core/kernels:quantized_resize_bilinear_op_test PASSED in 1.8s //tensorflow/core/kernels:ragged_fill_empty_rows_op_test PASSED in 0.6s //tensorflow/core/kernels:ragged_gather_op_test PASSED in 1.1s //tensorflow/core/kernels:ragged_range_op_test PASSED in 0.7s //tensorflow/core/kernels:ragged_tensor_from_variant_op_test PASSED in 1.1s //tensorflow/core/kernels:ragged_tensor_to_sparse_kernel_test PASSED in 1.4s //tensorflow/core/kernels:ragged_tensor_to_tensor_op_test PASSED in 0.5s //tensorflow/core/kernels:ragged_tensor_to_variant_op_test PASSED in 0.7s //tensorflow/core/kernels:random_binomial_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:random_index_shuffle_test PASSED in 0.3s //tensorflow/core/kernels:random_op_test_cpu PASSED in 0.4s //tensorflow/core/kernels:random_poisson_op_test_cpu PASSED in 1.2s //tensorflow/core/kernels:range_sampler_test PASSED in 0.5s //tensorflow/core/kernels:reduction_ops_test_cpu PASSED in 0.6s //tensorflow/core/kernels:regex_replace_op_test PASSED in 1.0s //tensorflow/core/kernels:requantization_range_op_test PASSED in 0.8s //tensorflow/core/kernels:requantize_op_test PASSED in 0.7s //tensorflow/core/kernels:resource_ops_test PASSED in 0.8s //tensorflow/core/kernels:restore_op_test PASSED in 0.6s //tensorflow/core/kernels:restore_v2_op_test PASSED in 1.1s //tensorflow/core/kernels:reverse_op_test PASSED in 0.5s //tensorflow/core/kernels:roll_op_test PASSED in 1.0s //tensorflow/core/kernels:save_op_test PASSED in 1.0s //tensorflow/core/kernels:save_v2_op_test PASSED in 0.8s //tensorflow/core/kernels:scan_ops_test_cpu PASSED in 1.5s //tensorflow/core/kernels:scatter_nd_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:scatter_op_test PASSED in 1.3s //tensorflow/core/kernels:scoped_allocator_ops_test_cpu PASSED in 7.8s //tensorflow/core/kernels:sdca_ops_test PASSED in 1.2s //tensorflow/core/kernels:segment_reduction_ops_test PASSED in 0.5s //tensorflow/core/kernels:sendrecv_ops_test PASSED in 0.7s //tensorflow/core/kernels:sequence_ops_test PASSED in 0.8s //tensorflow/core/kernels:shape_ops_test PASSED in 1.8s //tensorflow/core/kernels:slice_op_test PASSED in 0.4s //tensorflow/core/kernels:spacetobatch_benchmark_test_cpu PASSED in 2.3s //tensorflow/core/kernels:sparse_add_op_test PASSED in 0.8s //tensorflow/core/kernels:sparse_dense_binary_op_shared_test PASSED in 0.9s //tensorflow/core/kernels:sparse_fill_empty_rows_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:sparse_matmul_op_test_cpu PASSED in 1.8s //tensorflow/core/kernels:sparse_reduce_sum_op_test PASSED in 0.6s //tensorflow/core/kernels:sparse_tensor_dense_matmul_op_test_cpu PASSED in 1.5s //tensorflow/core/kernels:sparse_to_dense_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:sparse_utils_test PASSED in 0.4s //tensorflow/core/kernels:sparse_xent_op_test_cpu PASSED in 1.3s //tensorflow/core/kernels:spectrogram_op_test_cpu PASSED in 3.4s //tensorflow/core/kernels:spectrogram_test PASSED in 0.2s //tensorflow/core/kernels:split_op_test_cpu PASSED in 1.7s //tensorflow/core/kernels:split_v_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:strided_slice_op_test PASSED in 0.7s //tensorflow/core/kernels:string_format_op_test PASSED in 0.6s //tensorflow/core/kernels:string_ngrams_op_test PASSED in 0.9s //tensorflow/core/kernels:string_split_op_test PASSED in 0.5s //tensorflow/core/kernels:substr_op_test PASSED in 1.0s //tensorflow/core/kernels:summary_audio_op_test PASSED in 0.8s //tensorflow/core/kernels:summary_image_op_test PASSED in 1.0s //tensorflow/core/kernels:summary_op_test PASSED in 0.7s //tensorflow/core/kernels:summary_tensor_op_test PASSED in 0.7s //tensorflow/core/kernels:tensor_cord_test PASSED in 0.3s //tensorflow/core/kernels:tensor_flag_utils_test PASSED in 0.1s //tensorflow/core/kernels:tensor_map_test PASSED in 0.1s //tensorflow/core/kernels:training_ops_test PASSED in 0.5s //tensorflow/core/kernels:transpose_util_test PASSED in 0.3s //tensorflow/core/kernels:unary_ops_composition_test_cpu PASSED in 1.8s //tensorflow/core/kernels:unique_op_test PASSED in 0.5s //tensorflow/core/kernels:variable_ops_test PASSED in 1.4s //tensorflow/core/kernels:while_op_test PASSED in 0.9s //tensorflow/core/kernels:xent_op_test_cpu PASSED in 1.1s //tensorflow/core/kernels/batching_util:basic_batch_scheduler_test PASSED in 0.3s //tensorflow/core/kernels/batching_util:batch_input_task_test PASSED in 0.6s //tensorflow/core/kernels/batching_util:batch_resource_base_test PASSED in 0.1s //tensorflow/core/kernels/batching_util:batch_scheduler_test PASSED in 0.2s //tensorflow/core/kernels/batching_util:bounded_executor_test PASSED in 22.3s //tensorflow/core/kernels/batching_util:input_split_metadata_test PASSED in 0.1s //tensorflow/core/kernels/batching_util:periodic_function_test PASSED in 3.0s //tensorflow/core/kernels/batching_util:serial_device_batch_scheduler_test PASSED in 1.8s //tensorflow/core/kernels/batching_util:shared_batch_scheduler_test PASSED in 7.0s //tensorflow/core/kernels/batching_util:threadsafe_status_test PASSED in 0.1s //tensorflow/core/kernels/data:batch_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:cache_dataset_ops_test PASSED in 0.7s //tensorflow/core/kernels/data:concatenate_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:filter_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:finalize_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data:fixed_length_record_dataset_op_test PASSED in 2.0s //tensorflow/core/kernels/data:flat_map_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:get_options_op_test PASSED in 0.5s //tensorflow/core/kernels/data:interleave_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data:iterator_ops_test PASSED in 0.9s //tensorflow/core/kernels/data:map_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data:map_defun_op_test PASSED in 0.6s //tensorflow/core/kernels/data:optimize_dataset_op_test PASSED in 1.5s //tensorflow/core/kernels/data:options_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:padded_batch_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:parallel_batch_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:parallel_filter_dataset_op_test PASSED in 1.7s //tensorflow/core/kernels/data:parallel_interleave_dataset_op_test PASSED in 2.3s //tensorflow/core/kernels/data:parallel_map_dataset_op_test PASSED in 2.1s //tensorflow/core/kernels/data:prefetch_autotuner_test PASSED in 0.1s //tensorflow/core/kernels/data:prefetch_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data:range_dataset_op_test PASSED in 1.8s //tensorflow/core/kernels/data:reduce_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data:repeat_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:rewrite_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data:shard_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:shuffle_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:skip_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:sparse_tensor_slice_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:take_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:tensor_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:tensor_slice_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:text_line_dataset_op_test PASSED in 1.5s //tensorflow/core/kernels/data:tf_record_dataset_op_test PASSED in 2.3s //tensorflow/core/kernels/data:window_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:zip_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data/experimental:assert_next_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data/experimental:assert_prev_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data/experimental:auto_shard_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data/experimental:directed_interleave_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data/experimental:list_dataset_op_test PASSED in 0.5s //tensorflow/core/kernels/data/experimental:map_and_batch_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data/experimental:parallel_interleave_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data/experimental:random_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data/experimental:sampling_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data/experimental:save_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data/experimental:unique_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/image:adjust_contrast_op_benchmark_test_cpu PASSED in 0.6s //tensorflow/core/kernels/image:adjust_contrast_op_test PASSED in 0.7s //tensorflow/core/kernels/image:colorspace_op_test PASSED in 2.2s //tensorflow/core/kernels/image:crop_and_resize_op_benchmark_test_cpu PASSED in 1.0s //tensorflow/core/kernels/image:crop_and_resize_op_test PASSED in 0.9s //tensorflow/core/kernels/image:encode_jpeg_op_test PASSED in 0.7s //tensorflow/core/kernels/image:mirror_pad_op_benchmark_test_cpu PASSED in 0.6s //tensorflow/core/kernels/image:mirror_pad_op_test PASSED in 0.7s //tensorflow/core/kernels/image:non_max_suppression_op_benchmark_test PASSED in 0.5s //tensorflow/core/kernels/image:non_max_suppression_op_test PASSED in 2.9s //tensorflow/core/kernels/image:resize_area_op_test PASSED in 1.4s //tensorflow/core/kernels/image:resize_benchmark_test_cpu PASSED in 0.6s //tensorflow/core/kernels/image:resize_bicubic_op_test PASSED in 4.9s //tensorflow/core/kernels/image:resize_ops_test_cpu PASSED in 2.1s //tensorflow/core/kernels/image:sampling_kernels_test PASSED in 0.7s //tensorflow/core/kernels/image:scale_and_translate_op_test PASSED in 2.4s //tensorflow/core/kernels/linalg:banded_triangular_solve_op_test_cpu PASSED in 1.1s //tensorflow/core/kernels/linalg:matrix_triangular_solve_op_test_cpu PASSED in 1.3s //tensorflow/core/kernels/mkl:mkl_conv_ops_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_dequantize_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_fused_batch_norm_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_fused_ops_test PASSED in 0.6s //tensorflow/core/kernels/mkl:mkl_matmul_op_benchmark PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_qmatmul_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_quantize_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_quantized_concat_op_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_perchannel_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_test PASSED in 0.4s //tensorflow/core/kernels/mkl:mkl_quantized_pooling_ops_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_relu_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_requantize_ops_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_swish_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:onednn_nn_ops_benchmark PASSED in 0.1s //tensorflow/core/kernels/sparse:kernels_test PASSED in 0.6s //tensorflow/core/kernels/uniform_quant_ops:math_utils_test PASSED in 0.1s //tensorflow/core/kernels/uniform_quant_ops:tensor_utils_test PASSED in 0.1s //tensorflow/core/kernels/uniform_quant_ops:uniform_dequantize_op_test PASSED in 0.8s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantize_op_test PASSED in 0.7s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_add_op_test PASSED in 0.8s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_clip_by_value_op_test PASSED in 0.8s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_convolution_ops_test PASSED in 0.8s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_dot_ops_test PASSED in 1.0s //tensorflow/core/kernels/uniform_quant_ops:uniform_requantize_op_test PASSED in 1.0s //tensorflow/core/lib/db:sqlite_test PASSED in 0.3s //tensorflow/core/lib/gif:lib_gif_io_test PASSED in 0.9s //tensorflow/core/lib/jpeg:lib_jpeg_jpeg_mem_unittest PASSED in 0.8s //tensorflow/core/ops:cudnn_rnn_ops_test_cc PASSED in 1.2s //tensorflow/core/ops:ops_array_grad_test PASSED in 2.1s //tensorflow/core/ops:ops_math_grad_test PASSED in 4.5s //tensorflow/core/ops:ops_tests PASSED in 1.4s //tensorflow/core/ops/compat:backwards_compatibility_test PASSED in 2.1s //tensorflow/core/platform:__tensorflow_tsl_platform_profile_utils_cpu_utils_test PASSED in 0.2s //tensorflow/core/platform:enable_tf2_utils_test PASSED in 0.4s //tensorflow/core/platform:env_test PASSED in 4.3s //tensorflow/core/platform:fake_python_env_test PASSED in 0.4s //tensorflow/core/platform:file_system_test PASSED in 0.1s //tensorflow/core/platform:platform_strings_test PASSED in 0.1s //tensorflow/core/platform:ram_file_system_test PASSED in 39.1s //tensorflow/core/platform:resource_loader_test PASSED in 0.1s //tensorflow/core/platform:vmodule_benchmark_test PASSED in 0.1s //tensorflow/core/platform:vmodule_test PASSED in 0.7s //tensorflow/core/profiler/backends/cpu:host_tracer_test PASSED in 0.9s //tensorflow/core/profiler/convert:dcn_analysis_test PASSED in 0.3s //tensorflow/core/profiler/convert:dcn_utils_test PASSED in 0.5s //tensorflow/core/profiler/convert:hlo_proto_to_graph_view_test PASSED in 0.1s //tensorflow/core/profiler/convert:hlo_proto_to_memory_visualization_utils_test PASSED in 1.7s //tensorflow/core/profiler/convert:op_stats_to_pod_stats_test PASSED in 0.2s //tensorflow/core/profiler/convert:op_stats_to_pod_viewer_test PASSED in 0.3s //tensorflow/core/profiler/convert:op_stats_to_tf_stats_test PASSED in 0.7s //tensorflow/core/profiler/convert:repository_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_kernel_stats_db_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_memory_profile_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_op_metrics_db_test PASSED in 0.4s //tensorflow/core/profiler/convert:xplane_to_op_stats_test PASSED in 0.4s //tensorflow/core/profiler/convert:xplane_to_step_events_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_tf_functions_test PASSED in 0.4s //tensorflow/core/profiler/convert:xplane_to_tool_names_test PASSED in 0.2s //tensorflow/core/profiler/convert/trace_viewer:trace_viewer_visibility_test PASSED in 0.2s //tensorflow/core/profiler/internal:tfprof_show_test PASSED in 1.2s //tensorflow/core/profiler/internal:tfprof_stats_test PASSED in 1.4s //tensorflow/core/profiler/internal:tfprof_tensor_test PASSED in 14.5s //tensorflow/core/profiler/internal:tfprof_timeline_test PASSED in 2.0s //tensorflow/core/profiler/internal/advisor:tfprof_advisor_test PASSED in 0.8s //tensorflow/core/profiler/lib:profiler_disabled_test PASSED in 0.3s //tensorflow/core/profiler/utils:derived_timeline_test PASSED in 0.2s //tensorflow/core/profiler/utils:kernel_stats_utils_test PASSED in 0.1s //tensorflow/core/profiler/utils:op_metrics_db_utils_test PASSED in 0.2s //tensorflow/core/profiler/utils:step_intersection_test PASSED in 0.1s //tensorflow/core/runtime_fallback/util:type_util_test PASSED in 0.1s //tensorflow/core/summary:schema_test PASSED in 0.6s //tensorflow/core/summary:summary_db_writer_test PASSED in 0.1s //tensorflow/core/summary:summary_file_writer_test PASSED in 0.1s //tensorflow/core/tfrt/common:pjrt_cpu_client_registration_test PASSED in 18.7s //tensorflow/core/tfrt/common:pjrt_state_test PASSED in 7.9s //tensorflow/core/tfrt/common:pjrt_util_test PASSED in 6.0s //tensorflow/core/tfrt/fallback:cost_recorder_test PASSED in 0.1s //tensorflow/core/tfrt/fallback:fallback_state_test PASSED in 0.8s //tensorflow/core/tfrt/graph_executor:config_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/attribute:attribute_test PASSED in 1.1s //tensorflow/core/tfrt/mlrt/bytecode:bytecode_test PASSED in 0.2s //tensorflow/core/tfrt/mlrt/bytecode:executable_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/bytecode:function_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/bytecode:kernel_test PASSED in 0.5s //tensorflow/core/tfrt/mlrt/bytecode:span_test PASSED in 0.3s //tensorflow/core/tfrt/mlrt/interpreter:context_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/interpreter:future_test PASSED in 0.2s //tensorflow/core/tfrt/mlrt/interpreter:interpreter_test PASSED in 0.7s //tensorflow/core/tfrt/mlrt/interpreter:register_span_test PASSED in 0.1s //tensorflow/core/tfrt/mlrt/interpreter:value_test PASSED in 0.1s //tensorflow/core/tfrt/run_handler_thread_pool:run_handler_concurrent_work_queue_test PASSED in 1.3s //tensorflow/core/tfrt/run_handler_thread_pool:run_handler_test PASSED in 1.6s //tensorflow/core/tfrt/run_handler_thread_pool:run_handler_util_test PASSED in 0.4s //tensorflow/core/tfrt/runtime:channel_test PASSED in 0.2s //tensorflow/core/tfrt/runtime:tf_threadpool_concurrent_work_queue_test PASSED in 0.6s //tensorflow/core/tfrt/runtime:work_queue_interface_test PASSED in 0.7s //tensorflow/core/tfrt/utils:graph_partition_test PASSED in 2.2s //tensorflow/core/transforms:eval_utils_test PASSED in 2.6s //tensorflow/core/transforms:graph_transform_wrapper_test PASSED in 0.3s //tensorflow/core/util:bcast_test PASSED in 0.9s //tensorflow/core/util:command_line_flags_test PASSED in 2.4s //tensorflow/core/util:debug_data_dumper_test PASSED in 0.7s //tensorflow/core/util:debug_events_writer_test PASSED in 0.2s //tensorflow/core/util:dump_graph_test PASSED in 1.0s //tensorflow/core/util:equal_graph_def_test PASSED in 0.9s //tensorflow/core/util:events_writer_test PASSED in 2.9s //tensorflow/core/util:example_proto_fast_parsing_test PASSED in 1.3s //tensorflow/core/util:example_proto_helper_test PASSED in 0.8s //tensorflow/core/util:exec_on_stall_test PASSED in 2.1s //tensorflow/core/util:fake_clock_env_test PASSED in 1.8s //tensorflow/core/util:incremental_barrier_test PASSED in 0.1s //tensorflow/core/util:matmul_bcast_test PASSED in 1.0s //tensorflow/core/util:memmapped_file_system_test PASSED in 0.9s //tensorflow/core/util:overflow_test PASSED in 1.1s //tensorflow/core/util:presized_cuckoo_map_test PASSED in 2.6s //tensorflow/core/util:ragged_to_dense_util_test PASSED in 0.7s //tensorflow/core/util:reffed_status_callback_test PASSED in 2.1s //tensorflow/core/util:reporter_test PASSED in 1.0s //tensorflow/core/util:saved_tensor_slice_util_test PASSED in 0.8s //tensorflow/core/util:semver_test PASSED in 0.8s //tensorflow/core/util:stat_summarizer_test PASSED in 0.9s //tensorflow/core/util:strided_slice_op_test PASSED in 1.0s //tensorflow/core/util:tensor_format_test PASSED in 0.7s //tensorflow/core/util:tensor_slice_reader_test PASSED in 0.8s //tensorflow/core/util:tensor_slice_set_test PASSED in 1.4s //tensorflow/core/util:tensor_slice_util_test PASSED in 0.6s //tensorflow/core/util:tensor_slice_writer_test PASSED in 1.4s //tensorflow/core/util:work_sharder_test PASSED in 0.9s //tensorflow/core/util/ctc:ctc_beam_search_test PASSED in 0.1s //tensorflow/core/util/proto:descriptor_pool_registry_test PASSED in 0.5s //tensorflow/core/util/proto:proto_utils_test PASSED in 0.5s //tensorflow/core/util/quantization:uniform_quant_ops_params_test PASSED in 0.7s //tensorflow/core/util/sparse:sparse_tensor_test PASSED in 0.1s //tensorflow/core/util/tensor_bundle:tensor_bundle_test PASSED in 33.3s //tensorflow/dtensor/mlir:dtensor_location_test PASSED in 1.7s //tensorflow/dtensor/mlir:group_assignment_test PASSED in 0.2s //tensorflow/dtensor/mlir/tests:annotate_global_shape.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:cluster_function_conversion.mlir.test PASSED in 1.8s //tensorflow/dtensor/mlir/tests:constant_folding.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:decompose_controlflow.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:designate_resource_handle_mesh.mlir.test PASSED in 3.4s //tensorflow/dtensor/mlir/tests:device_mesh_cluster_coarsening.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_all_gather.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_all_scatter.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_combine_optimization.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_lowering.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_scatter_optimization.mlir.test PASSED in 1.2s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_sum_optimization.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_alltoall_lowering.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:dtensor_collective_type_lowering.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:dtensor_layout_must_execute.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:dtensor_layout_to_xla_sharding_op.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:dtensor_mixed_precision_reduce.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_reduce_scatter_lowering.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_remove_dtensorlayout.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:dtensor_replace_auxiliary_layout_op.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:dtensor_replace_relayout_with_identity.mlir.test PASSED in 2.0s //tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding_default.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_xla_spmd_integration.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:elide_identity_before_copy_to_mesh.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:function_renaming.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:handle_cross_cluster_dependencies.mlir.test PASSED in 2.2s //tensorflow/dtensor/mlir/tests:handle_sparsetensors.mlir.test PASSED in 11.3s //tensorflow/dtensor/mlir/tests:layout_propagation_v2.mlir.test PASSED in 1.1s //tensorflow/dtensor/mlir/tests:lower_send_recv.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:merge_clusters.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:mesh_propagation.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:multi_device_expansion.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:op_to_device_cluster.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:propagate_default_layout.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:propagate_device_id_to_function.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:restore_and_assign.mlir.test PASSED in 2.7s //tensorflow/dtensor/mlir/tests:restore_shape_inference.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:set_default_sharding.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:sparse_expansion.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_batchparallel.mlir.test PASSED in 1.3s //tensorflow/dtensor/mlir/tests:spmd_concat.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:spmd_conv.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:spmd_einsum.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_expansion.mlir.test PASSED in 1.3s //tensorflow/dtensor/mlir/tests:spmd_io_ops.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:spmd_iterator.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_matmul.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_random.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_save_restore.mlir.test PASSED in 1.2s //tensorflow/dtensor/mlir/tests:spmd_segment_sum.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_slice.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_softmax_loss.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:spmd_squeeze.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:spmd_var_handle.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:tf_dtensor_ops.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:tpu_add_resource_device_attribute.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:tpu_integration.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:undo_merge_const_across_mesh.mlir.test PASSED in 1.9s //tensorflow/dtensor/mlir/tests:update_tpu_metadata.mlir.test PASSED in 0.8s //tensorflow/dtensor/python/tests:array_ops_test_cpu PASSED in 24.7s //tensorflow/dtensor/python/tests:collective_combine_all_reduce_test_cpu PASSED in 21.9s //tensorflow/dtensor/python/tests:collective_test_cpu PASSED in 20.8s //tensorflow/dtensor/python/tests:config_test_cpu PASSED in 9.8s //tensorflow/dtensor/python/tests:device_test_cpu PASSED in 59.2s //tensorflow/dtensor/python/tests:layout_test_cpu PASSED in 27.8s //tensorflow/dtensor/python/tests:multi_client_test_cpu PASSED in 18.2s //tensorflow/dtensor/python/tests:numpy_util_test_cpu PASSED in 11.4s //tensorflow/dtensor/tests:dtensor_operation_test PASSED in 32.7s //tensorflow/dtensor/tests:executable_manager_test PASSED in 31.3s //tensorflow/dtensor/tests:layout_to_xla_sharding_test PASSED in 0.2s //tensorflow/dtensor/tests:slice_util_test PASSED in 0.1s //tensorflow/dtensor/tests:spmd_expander_test PASSED in 8.9s //tensorflow/dtensor/tests:tensor_layout_test PASSED in 0.1s //tensorflow/examples/adding_an_op:fact_test PASSED in 30.4s //tensorflow/examples/adding_an_op:zero_out_1_test PASSED in 32.7s //tensorflow/examples/adding_an_op:zero_out_2_test PASSED in 21.3s //tensorflow/examples/adding_an_op:zero_out_3_test PASSED in 22.3s //tensorflow/examples/custom_ops_doc/multiplex_1:multiplex_1_test PASSED in 64.8s //tensorflow/examples/custom_ops_doc/multiplex_2:multiplex_2_test_cpu PASSED in 26.6s //tensorflow/examples/custom_ops_doc/multiplex_3:multiplex_3_test PASSED in 38.8s //tensorflow/examples/custom_ops_doc/multiplex_4:multiplex_4_test PASSED in 22.9s //tensorflow/examples/custom_ops_doc/simple_hash_table:simple_hash_table_test PASSED in 26.7s //tensorflow/examples/custom_ops_doc/sleep:sleep_test PASSED in 38.3s //tensorflow/examples/speech_commands:accuracy_utils_test PASSED in 3.0s //tensorflow/examples/speech_commands:models_test PASSED in 27.6s //tensorflow/examples/speech_commands:recognize_commands_test PASSED in 1.5s //tensorflow/examples/wav_to_spectrogram:wav_to_spectrogram_test PASSED in 1.6s //tensorflow/js:ts_op_gen_test PASSED in 0.2s //tensorflow/python/autograph/converters:asserts_test PASSED in 26.8s //tensorflow/python/autograph/converters:break_statements_test PASSED in 13.9s //tensorflow/python/autograph/converters:call_trees_test PASSED in 12.5s //tensorflow/python/autograph/converters:conditional_expressions_test PASSED in 10.2s //tensorflow/python/autograph/converters:continue_statements_test PASSED in 12.8s //tensorflow/python/autograph/converters:control_flow_test PASSED in 17.0s //tensorflow/python/autograph/converters:directives_test PASSED in 12.1s //tensorflow/python/autograph/converters:functions_test PASSED in 11.1s //tensorflow/python/autograph/converters:lists_test PASSED in 9.1s //tensorflow/python/autograph/converters:logical_expressions_test PASSED in 11.0s //tensorflow/python/autograph/converters:return_statements_test PASSED in 9.7s //tensorflow/python/autograph/converters:slices_test PASSED in 9.1s //tensorflow/python/autograph/converters:variables_test PASSED in 10.9s //tensorflow/python/autograph/core:converter_test PASSED in 18.0s //tensorflow/python/autograph/core:function_wrappers_test PASSED in 12.5s //tensorflow/python/autograph/impl:api_test PASSED in 20.4s //tensorflow/python/autograph/impl:conversion_test PASSED in 11.7s //tensorflow/python/autograph/lang:special_functions_test PASSED in 25.6s //tensorflow/python/autograph/operators:conditional_expressions_test PASSED in 12.1s //tensorflow/python/autograph/operators:control_flow_test PASSED in 24.4s //tensorflow/python/autograph/operators:data_structures_test PASSED in 12.8s //tensorflow/python/autograph/operators:exceptions_test PASSED in 10.7s //tensorflow/python/autograph/operators:logical_test PASSED in 13.8s //tensorflow/python/autograph/operators:py_builtins_test PASSED in 34.9s //tensorflow/python/autograph/operators:slices_test PASSED in 18.0s //tensorflow/python/autograph/operators:variables_test PASSED in 9.2s //tensorflow/python/autograph/pyct:anno_test PASSED in 9.6s //tensorflow/python/autograph/pyct:ast_util_test PASSED in 12.0s //tensorflow/python/autograph/pyct:cache_test PASSED in 8.3s //tensorflow/python/autograph/pyct:cfg_test PASSED in 8.8s //tensorflow/python/autograph/pyct:error_utils_test PASSED in 9.6s //tensorflow/python/autograph/pyct:inspect_utils_test PASSED in 15.2s //tensorflow/python/autograph/pyct:loader_test PASSED in 8.2s //tensorflow/python/autograph/pyct:naming_test PASSED in 11.8s //tensorflow/python/autograph/pyct:origin_info_test PASSED in 11.4s //tensorflow/python/autograph/pyct:parser_test PASSED in 12.7s //tensorflow/python/autograph/pyct:pretty_printer_test PASSED in 11.4s //tensorflow/python/autograph/pyct:qual_names_test PASSED in 9.1s //tensorflow/python/autograph/pyct:templates_test PASSED in 11.5s //tensorflow/python/autograph/pyct:transformer_test PASSED in 11.5s //tensorflow/python/autograph/pyct:transpiler_test PASSED in 11.6s //tensorflow/python/autograph/pyct/static_analysis:activity_test PASSED in 27.0s //tensorflow/python/autograph/pyct/static_analysis:liveness_test PASSED in 10.4s //tensorflow/python/autograph/pyct/static_analysis:reaching_definitions_test PASSED in 8.6s //tensorflow/python/autograph/pyct/static_analysis:reaching_fndefs_test PASSED in 10.4s //tensorflow/python/autograph/pyct/static_analysis:type_inference_test PASSED in 11.2s //tensorflow/python/autograph/tests:assertion_test PASSED in 22.1s //tensorflow/python/autograph/tests:basic_ifexp_test PASSED in 47.1s //tensorflow/python/autograph/tests:call_to_builtin_function_test PASSED in 23.6s //tensorflow/python/autograph/tests:call_to_lambda_function_test PASSED in 24.2s //tensorflow/python/autograph/tests:call_to_named_tuple_test PASSED in 23.7s //tensorflow/python/autograph/tests:call_to_numpy_function_test PASSED in 28.4s //tensorflow/python/autograph/tests:call_to_print_function_test PASSED in 36.1s //tensorflow/python/autograph/tests:call_to_tf_api_test PASSED in 20.7s //tensorflow/python/autograph/tests:call_to_user_function_test PASSED in 26.2s //tensorflow/python/autograph/tests:composite_names_in_control_flow_test PASSED in 33.0s //tensorflow/python/autograph/tests:cond_basic_test PASSED in 33.4s //tensorflow/python/autograph/tests:datasets_test PASSED in 34.6s //tensorflow/python/autograph/tests:early_return_test PASSED in 28.3s //tensorflow/python/autograph/tests:ext_slice_test PASSED in 25.7s //tensorflow/python/autograph/tests:generator_test PASSED in 26.1s //tensorflow/python/autograph/tests:logical_expression_test PASSED in 23.4s //tensorflow/python/autograph/tests:loop_basic_test PASSED in 112.7s //tensorflow/python/autograph/tests:loop_control_flow_illegal_cases_test PASSED in 29.1s //tensorflow/python/autograph/tests:loop_created_variables_test PASSED in 30.7s //tensorflow/python/autograph/tests:loop_scoping_test PASSED in 45.8s //tensorflow/python/autograph/tests:loop_with_function_call_test PASSED in 30.8s //tensorflow/python/autograph/tests:loop_with_variable_type_illegal_cases_test PASSED in 28.8s //tensorflow/python/autograph/tests:loop_with_variable_type_test PASSED in 56.2s //tensorflow/python/autograph/tests:nested_control_flow_test PASSED in 56.0s //tensorflow/python/autograph/tests:type_annotations_test PASSED in 21.9s //tensorflow/python/autograph/utils:context_managers_test PASSED in 9.9s //tensorflow/python/autograph/utils:misc_test PASSED in 11.5s //tensorflow/python/autograph/utils:tensor_list_test PASSED in 21.1s //tensorflow/python/autograph/utils:tensors_test PASSED in 11.1s //tensorflow/python/checkpoint:benchmarks_test PASSED in 12.7s //tensorflow/python/checkpoint:checkpoint_management_test_cpu PASSED in 18.3s //tensorflow/python/checkpoint:checkpoint_metrics_test PASSED in 17.5s //tensorflow/python/checkpoint:checkpoint_test PASSED in 28.2s //tensorflow/python/checkpoint:checkpoint_view_test PASSED in 26.0s //tensorflow/python/checkpoint:checkpoint_with_v1_optimizers_test PASSED in 12.6s //tensorflow/python/checkpoint:functional_saver_test_cpu PASSED in 10.9s //tensorflow/python/checkpoint:restore_test PASSED in 16.4s //tensorflow/python/checkpoint:save_util_v1_test PASSED in 13.5s //tensorflow/python/checkpoint:saveable_compat_test PASSED in 13.0s //tensorflow/python/checkpoint:tensor_callable_test PASSED in 18.5s //tensorflow/python/checkpoint:trackable_view_test PASSED in 11.3s //tensorflow/python/client:device_lib_test_cpu PASSED in 12.5s //tensorflow/python/client:events_writer_test PASSED in 9.9s //tensorflow/python/client:session_benchmark_cpu PASSED in 9.3s //tensorflow/python/client:session_list_devices_test PASSED in 12.0s //tensorflow/python/client:session_partial_run_test PASSED in 13.1s //tensorflow/python/client:timeline_test_cpu PASSED in 26.8s //tensorflow/python/client:virtual_gpu_test_cpu PASSED in 10.5s //tensorflow/python/compat:compat_test PASSED in 9.6s //tensorflow/python/compat:disable_v2_behavior_test PASSED in 10.6s //tensorflow/python/compiler/mlir:mlir_test PASSED in 9.6s //tensorflow/python/compiler/tensorrt:trt_convert_test_cpu PASSED in 11.8s //tensorflow/python/compiler/tensorrt/test:batch_matmul_test_cpu PASSED in 11.3s //tensorflow/python/compiler/tensorrt/test:biasadd_matmul_test_cpu PASSED in 12.0s //tensorflow/python/compiler/tensorrt/test:binary_tensor_weight_broadcast_test_cpu PASSED in 13.5s //tensorflow/python/compiler/tensorrt/test:bool_test_cpu PASSED in 20.5s //tensorflow/python/compiler/tensorrt/test:cast_test_cpu PASSED in 12.6s //tensorflow/python/compiler/tensorrt/test:concatenation_test_cpu PASSED in 25.1s //tensorflow/python/compiler/tensorrt/test:const_broadcast_test_cpu PASSED in 14.6s //tensorflow/python/compiler/tensorrt/test:data_dependent_shape_test_cpu PASSED in 25.5s //tensorflow/python/compiler/tensorrt/test:dynamic_input_shapes_test_cpu PASSED in 10.5s //tensorflow/python/compiler/tensorrt/test:identity_output_test_cpu PASSED in 28.4s //tensorflow/python/compiler/tensorrt/test:int32_test_cpu PASSED in 10.3s //tensorflow/python/compiler/tensorrt/test:lru_cache_test_cpu PASSED in 14.7s //tensorflow/python/compiler/tensorrt/test:multi_connection_neighbor_engine_test_cpu PASSED in 11.8s //tensorflow/python/compiler/tensorrt/test:neighboring_engine_test_cpu PASSED in 13.4s //tensorflow/python/compiler/tensorrt/test:quantization_test_cpu PASSED in 11.6s //tensorflow/python/compiler/tensorrt/test:rank_two_test_cpu PASSED in 11.0s //tensorflow/python/compiler/tensorrt/test:reshape_transpose_test_cpu PASSED in 12.0s //tensorflow/python/compiler/tensorrt/test:topk_test_cpu PASSED in 18.4s //tensorflow/python/compiler/tensorrt/test:trt_engine_op_shape_test_cpu PASSED in 14.1s //tensorflow/python/compiler/tensorrt/test:trt_mode_test_cpu PASSED in 13.1s //tensorflow/python/compiler/tensorrt/test:unary_test_cpu PASSED in 10.7s //tensorflow/python/compiler/tensorrt/test:vgg_block_nchw_test_cpu PASSED in 12.3s //tensorflow/python/compiler/tensorrt/test:vgg_block_test_cpu PASSED in 27.0s //tensorflow/python/compiler/xla:jit_compile_test_cpu PASSED in 16.9s //tensorflow/python/compiler/xla:jit_test_cpu PASSED in 17.3s //tensorflow/python/compiler/xla:xla_test_cpu PASSED in 24.8s //tensorflow/python/compiler/xla/experimental:xla_sharding_test PASSED in 9.7s //tensorflow/python/data/benchmarks:batch_benchmark PASSED in 13.5s //tensorflow/python/data/benchmarks:filter_benchmark PASSED in 12.6s //tensorflow/python/data/benchmarks:from_tensor_slices_benchmark PASSED in 10.4s //tensorflow/python/data/benchmarks:interleave_benchmark PASSED in 10.8s //tensorflow/python/data/benchmarks:list_files_benchmark PASSED in 11.5s //tensorflow/python/data/benchmarks:map_benchmark PASSED in 11.2s //tensorflow/python/data/benchmarks:meta_benchmark PASSED in 13.4s //tensorflow/python/data/benchmarks:prefetch_benchmark PASSED in 10.7s //tensorflow/python/data/benchmarks:range_benchmark PASSED in 13.5s //tensorflow/python/data/experimental/benchmarks:autotune_benchmark PASSED in 11.8s //tensorflow/python/data/experimental/benchmarks:csv_dataset_benchmark PASSED in 21.9s //tensorflow/python/data/experimental/benchmarks:map_and_batch_benchmark PASSED in 10.0s //tensorflow/python/data/experimental/benchmarks:map_defun_benchmark PASSED in 10.6s //tensorflow/python/data/experimental/benchmarks:matching_files_benchmark PASSED in 26.8s //tensorflow/python/data/experimental/benchmarks:optimize_benchmark PASSED in 12.2s //tensorflow/python/data/experimental/benchmarks:parameter_value_benchmark PASSED in 9.2s //tensorflow/python/data/experimental/benchmarks:rejection_resample_benchmark PASSED in 9.9s //tensorflow/python/data/experimental/benchmarks:snapshot_dataset_benchmark PASSED in 11.4s //tensorflow/python/data/experimental/benchmarks:unbatch_benchmark PASSED in 12.6s //tensorflow/python/data/experimental/kernel_tests:assert_cardinality_test PASSED in 42.3s //tensorflow/python/data/experimental/kernel_tests:assert_next_test PASSED in 12.1s //tensorflow/python/data/experimental/kernel_tests:assert_prev_test PASSED in 14.7s //tensorflow/python/data/experimental/kernel_tests:checkpoint_input_pipeline_hook_test PASSED in 21.3s //tensorflow/python/data/experimental/kernel_tests:compression_ops_test PASSED in 17.5s //tensorflow/python/data/experimental/kernel_tests:copy_to_device_test_cpu PASSED in 22.1s //tensorflow/python/data/experimental/kernel_tests:dense_to_sparse_batch_test PASSED in 24.4s //tensorflow/python/data/experimental/kernel_tests:from_list_test PASSED in 32.9s //tensorflow/python/data/experimental/kernel_tests:io_test PASSED in 33.7s //tensorflow/python/data/experimental/kernel_tests:lookup_ops_test PASSED in 12.9s //tensorflow/python/data/experimental/kernel_tests:make_csv_dataset_test PASSED in 28.1s //tensorflow/python/data/experimental/kernel_tests:make_saveable_from_iterator_test PASSED in 11.9s //tensorflow/python/data/experimental/kernel_tests:make_tf_record_dataset_test PASSED in 61.1s //tensorflow/python/data/experimental/kernel_tests:map_defun_op_test PASSED in 10.4s //tensorflow/python/data/experimental/kernel_tests:matching_files_dataset_test PASSED in 27.0s //tensorflow/python/data/experimental/kernel_tests:model_dataset_test PASSED in 11.2s //tensorflow/python/data/experimental/kernel_tests:non_serializable_test PASSED in 15.1s //tensorflow/python/data/experimental/kernel_tests:pad_to_cardinality_test PASSED in 16.0s //tensorflow/python/data/experimental/kernel_tests:prefetch_to_device_test_cpu PASSED in 29.7s //tensorflow/python/data/experimental/kernel_tests:prefetch_with_slack_test PASSED in 14.4s //tensorflow/python/data/experimental/kernel_tests:shuffle_and_repeat_test PASSED in 26.4s //tensorflow/python/data/experimental/kernel_tests:sleep_test PASSED in 12.7s //tensorflow/python/data/experimental/kernel_tests:tf_record_writer_test PASSED in 13.5s //tensorflow/python/data/experimental/kernel_tests:variant_test PASSED in 12.4s //tensorflow/python/data/experimental/kernel_tests:wrap_unwrap_test_cpu PASSED in 10.8s //tensorflow/python/data/experimental/kernel_tests/optimization:filter_fusion_test PASSED in 37.1s //tensorflow/python/data/experimental/kernel_tests/optimization:filter_parallelization_test PASSED in 119.5s //tensorflow/python/data/experimental/kernel_tests/optimization:grappler_test_cpu PASSED in 20.9s //tensorflow/python/data/experimental/kernel_tests/optimization:make_deterministic_test PASSED in 39.5s //tensorflow/python/data/experimental/kernel_tests/optimization:map_and_batch_fusion_test PASSED in 11.4s //tensorflow/python/data/experimental/kernel_tests/optimization:map_and_filter_fusion_test PASSED in 20.7s //tensorflow/python/data/experimental/kernel_tests/optimization:map_fusion_test PASSED in 24.0s //tensorflow/python/data/experimental/kernel_tests/optimization:map_parallelization_test PASSED in 22.1s //tensorflow/python/data/experimental/kernel_tests/optimization:noop_elimination_test PASSED in 30.8s //tensorflow/python/data/experimental/kernel_tests/service:multi_device_test PASSED in 16.8s //tensorflow/python/data/experimental/service:server_lib_test PASSED in 9.4s //tensorflow/python/data/kernel_tests:as_numpy_iterator_test PASSED in 11.6s //tensorflow/python/data/kernel_tests:bucket_by_sequence_length_test PASSED in 24.4s //tensorflow/python/data/kernel_tests:cache_test PASSED in 47.6s //tensorflow/python/data/kernel_tests:cardinality_test PASSED in 27.9s //tensorflow/python/data/kernel_tests:checkpoint_test PASSED in 21.5s //tensorflow/python/data/kernel_tests:concatenate_test PASSED in 44.8s //tensorflow/python/data/kernel_tests:counter_test PASSED in 33.9s //tensorflow/python/data/kernel_tests:dataset_spec_test PASSED in 14.9s //tensorflow/python/data/kernel_tests:dataset_test PASSED in 38.5s //tensorflow/python/data/kernel_tests:enumerate_test PASSED in 26.4s //tensorflow/python/data/kernel_tests:from_sparse_tensor_slices_test PASSED in 10.5s //tensorflow/python/data/kernel_tests:from_tensor_slices_test PASSED in 31.4s //tensorflow/python/data/kernel_tests:from_tensors_test PASSED in 22.9s //tensorflow/python/data/kernel_tests:get_single_element_test PASSED in 25.5s //tensorflow/python/data/kernel_tests:ignore_errors_test PASSED in 30.9s //tensorflow/python/data/kernel_tests:io_test PASSED in 50.1s //tensorflow/python/data/kernel_tests:iterator_test_cpu PASSED in 28.3s //tensorflow/python/data/kernel_tests:len_test PASSED in 14.0s //tensorflow/python/data/kernel_tests:list_files_test PASSED in 25.0s //tensorflow/python/data/kernel_tests:optional_test_cpu PASSED in 14.7s //tensorflow/python/data/kernel_tests:options_test PASSED in 12.5s //tensorflow/python/data/kernel_tests:placement_test_cpu PASSED in 16.8s //tensorflow/python/data/kernel_tests:prefetch_test PASSED in 58.4s //tensorflow/python/data/kernel_tests:random_test PASSED in 53.1s //tensorflow/python/data/kernel_tests:range_test PASSED in 56.3s //tensorflow/python/data/kernel_tests:rebatch_test PASSED in 14.4s //tensorflow/python/data/kernel_tests:reduce_test_cpu PASSED in 28.0s //tensorflow/python/data/kernel_tests:scan_test_cpu PASSED in 43.7s //tensorflow/python/data/kernel_tests:sparse_batch_test PASSED in 18.9s //tensorflow/python/data/kernel_tests:unbatch_test PASSED in 43.8s //tensorflow/python/data/util:convert_test PASSED in 21.2s //tensorflow/python/data/util:nest_test PASSED in 10.7s //tensorflow/python/data/util:options_test PASSED in 9.1s //tensorflow/python/data/util:random_seed_test PASSED in 11.6s //tensorflow/python/data/util:sparse_test PASSED in 26.9s //tensorflow/python/data/util:structure_test PASSED in 11.4s //tensorflow/python/data/util:traverse_test PASSED in 27.0s //tensorflow/python/debug/cli:analyzer_cli_test_cpu PASSED in 12.8s //tensorflow/python/debug/cli:cli_config_test PASSED in 12.0s //tensorflow/python/debug/cli:cli_shared_test PASSED in 9.4s //tensorflow/python/debug/cli:command_parser_test PASSED in 9.7s //tensorflow/python/debug/cli:debugger_cli_common_test PASSED in 11.4s //tensorflow/python/debug/cli:evaluator_test PASSED in 11.7s //tensorflow/python/debug/cli:profile_analyzer_cli_test PASSED in 14.2s //tensorflow/python/debug/cli:readline_ui_test PASSED in 9.9s //tensorflow/python/debug/cli:tensor_format_test PASSED in 10.2s //tensorflow/python/debug/lib:check_numerics_callback_test_cpu PASSED in 13.7s //tensorflow/python/debug/lib:common_test PASSED in 14.3s //tensorflow/python/debug/lib:debug_data_test PASSED in 14.4s //tensorflow/python/debug/lib:debug_events_monitors_test PASSED in 11.1s //tensorflow/python/debug/lib:debug_events_writer_test PASSED in 10.8s //tensorflow/python/debug/lib:debug_gradients_test_cpu PASSED in 10.5s //tensorflow/python/debug/lib:debug_graph_reconstruction_test_cpu PASSED in 11.4s //tensorflow/python/debug/lib:debug_graphs_test PASSED in 8.8s //tensorflow/python/debug/lib:debug_grappler_test_cpu PASSED in 13.1s //tensorflow/python/debug/lib:debug_utils_test PASSED in 11.4s //tensorflow/python/debug/lib:debug_v2_ops_test_cpu PASSED in 16.4s //tensorflow/python/debug/lib:profiling_test PASSED in 9.4s //tensorflow/python/debug/lib:session_debug_file_test_cpu PASSED in 25.4s //tensorflow/python/debug/lib:session_debug_multi_gpu_test_cpu PASSED in 11.4s //tensorflow/python/debug/lib:source_utils_test PASSED in 12.5s //tensorflow/python/debug/wrappers:disk_usage_test PASSED in 10.9s //tensorflow/python/debug/wrappers:dumping_wrapper_test PASSED in 16.9s //tensorflow/python/debug/wrappers:framework_test PASSED in 22.5s //tensorflow/python/debug/wrappers:local_cli_wrapper_test PASSED in 10.2s //tensorflow/python/distribute:checkpoint_utils_test_2gpu PASSED in 14.8s //tensorflow/python/distribute:checkpoint_utils_test_cpu PASSED in 12.6s //tensorflow/python/distribute:checkpointing_test_2gpu PASSED in 11.5s //tensorflow/python/distribute:checkpointing_test_cpu PASSED in 21.7s //tensorflow/python/distribute:collective_util_test PASSED in 9.1s //tensorflow/python/distribute:combinations_test_2gpu PASSED in 40.2s //tensorflow/python/distribute:combinations_test_cpu PASSED in 40.8s //tensorflow/python/distribute:cross_device_utils_test_cpu PASSED in 15.3s //tensorflow/python/distribute:custom_training_loop_gradient_test_2gpu PASSED in 18.8s //tensorflow/python/distribute:custom_training_loop_gradient_test_cpu PASSED in 13.6s //tensorflow/python/distribute:device_util_test_cpu PASSED in 22.0s //tensorflow/python/distribute:distribute_coordinator_test PASSED in 21.1s //tensorflow/python/distribute:distribute_lib_test PASSED in 21.7s //tensorflow/python/distribute:distribute_utils_test_2gpu PASSED in 11.4s //tensorflow/python/distribute:distribute_utils_test_cpu PASSED in 22.1s //tensorflow/python/distribute:input_ops_test_cpu PASSED in 16.8s //tensorflow/python/distribute:metrics_v1_test_2gpu PASSED in 67.5s //tensorflow/python/distribute:metrics_v1_test_cpu PASSED in 39.2s //tensorflow/python/distribute:mirrored_values_test_2gpu PASSED in 11.7s //tensorflow/python/distribute:mirrored_values_test_cpu PASSED in 13.3s //tensorflow/python/distribute:mirrored_variable_test_2gpu PASSED in 21.7s //tensorflow/python/distribute:mirrored_variable_test_cpu PASSED in 32.7s //tensorflow/python/distribute:multi_process_runner_no_init_test PASSED in 13.7s //tensorflow/python/distribute:multi_worker_continuous_run_test_cpu PASSED in 23.3s //tensorflow/python/distribute:multi_worker_util_test PASSED in 11.1s //tensorflow/python/distribute:numpy_dataset_test PASSED in 13.6s //tensorflow/python/distribute:one_device_strategy_test_cpu PASSED in 27.9s //tensorflow/python/distribute:packed_distributed_variable_test PASSED in 11.1s //tensorflow/python/distribute:parameter_server_strategy_test_2gpu PASSED in 62.4s //tensorflow/python/distribute:parameter_server_strategy_test_cpu PASSED in 58.6s //tensorflow/python/distribute:parameter_server_strategy_v2_test_2gpu PASSED in 49.4s //tensorflow/python/distribute:parameter_server_strategy_v2_test_cpu PASSED in 34.4s //tensorflow/python/distribute:per_replica_test_2gpu PASSED in 10.8s //tensorflow/python/distribute:per_replica_test_cpu PASSED in 12.4s //tensorflow/python/distribute:ps_values_test_2gpu PASSED in 33.1s //tensorflow/python/distribute:ps_values_test_cpu PASSED in 9.1s //tensorflow/python/distribute:remote_mirrored_strategy_eager_test_cpu PASSED in 12.2s //tensorflow/python/distribute:sharded_variable_test PASSED in 29.5s //tensorflow/python/distribute:shared_variable_creator_test PASSED in 13.6s //tensorflow/python/distribute:strategy_combinations_test_cpu PASSED in 47.1s //tensorflow/python/distribute:template_mirrored_strategy_test_cpu PASSED in 10.7s //tensorflow/python/distribute:test_util_test_2gpu PASSED in 19.3s //tensorflow/python/distribute:test_util_test_cpu PASSED in 30.2s //tensorflow/python/distribute:tf_function_test_2gpu PASSED in 12.1s //tensorflow/python/distribute:tf_function_test_cpu PASSED in 11.9s //tensorflow/python/distribute:values_v2_test_cpu PASSED in 13.8s //tensorflow/python/distribute:warm_starting_util_test_2gpu PASSED in 16.0s //tensorflow/python/distribute:warm_starting_util_test_cpu PASSED in 13.9s //tensorflow/python/distribute/cluster_resolver:base_cluster_resolver_py_test PASSED in 9.5s //tensorflow/python/distribute/cluster_resolver:gce_cluster_resolver_py_test PASSED in 9.4s //tensorflow/python/distribute/cluster_resolver:kubernetes_cluster_resolver_py_test PASSED in 8.9s //tensorflow/python/distribute/cluster_resolver:sagemaker_cluster_resolver_py_test PASSED in 9.5s //tensorflow/python/distribute/cluster_resolver:slurm_cluster_resolver_py_test PASSED in 10.9s //tensorflow/python/distribute/cluster_resolver:tfconfig_cluster_resolver_py_test PASSED in 24.1s //tensorflow/python/distribute/cluster_resolver/tpu:tpu_cluster_resolver_py_test PASSED in 12.1s //tensorflow/python/distribute/coordinator:watchdog_test PASSED in 63.6s //tensorflow/python/distribute/experimental:dtensor_util_test_cpu PASSED in 16.8s //tensorflow/python/distribute/experimental:mirrored_strategy_test_cpu PASSED in 42.1s //tensorflow/python/distribute/experimental:multi_worker_mirrored_strategy_test_cpu PASSED in 25.5s //tensorflow/python/distribute/integration_test:saved_model_test_cpu PASSED in 64.7s //tensorflow/python/distribute/parallel_device:parallel_device_test_cpu PASSED in 14.1s //tensorflow/python/distribute/v1:all_reduce_test PASSED in 53.5s //tensorflow/python/distribute/v1:cross_device_ops_test_2gpu PASSED in 66.6s //tensorflow/python/distribute/v1:cross_device_ops_test_cpu PASSED in 57.4s //tensorflow/python/dlpack:dlpack_test_cpu PASSED in 14.3s //tensorflow/python/eager:backprop_test_cpu PASSED in 146.5s //tensorflow/python/eager:benchmarks_test_cpu PASSED in 10.7s //tensorflow/python/eager:cancellation_test_cpu PASSED in 10.9s //tensorflow/python/eager:context_test_cpu PASSED in 10.9s //tensorflow/python/eager:core_test_cpu PASSED in 22.7s //tensorflow/python/eager:gradient_input_output_exclusions_test PASSED in 58.7s //tensorflow/python/eager:graph_only_ops_test_cpu PASSED in 8.8s //tensorflow/python/eager:lift_to_graph_test PASSED in 11.3s //tensorflow/python/eager:monitoring_test_cpu PASSED in 14.2s //tensorflow/python/eager:ops_test_cpu PASSED in 14.0s //tensorflow/python/eager:profiler_client_test PASSED in 8.6s //tensorflow/python/eager:profiler_test_cpu PASSED in 9.7s //tensorflow/python/eager:pywrap_tfe_test PASSED in 30.6s //tensorflow/python/eager:record_test PASSED in 37.0s //tensorflow/python/eager:remote_benchmarks_test_cpu PASSED in 13.8s //tensorflow/python/eager:run_eager_op_as_function_test_cpu PASSED in 13.8s //tensorflow/python/eager:run_eager_op_as_function_xla_test_cpu PASSED in 15.1s //tensorflow/python/eager:small_constants_optimizer_test_cpu PASSED in 232.1s //tensorflow/python/eager:tensor_test_cpu PASSED in 15.2s //tensorflow/python/eager:wrap_function_device_test_cpu PASSED in 13.3s //tensorflow/python/eager:wrap_function_test PASSED in 14.2s //tensorflow/python/eager/benchmarks:kpi_benchmark_test_cpu PASSED in 24.5s //tensorflow/python/eager/memory_tests:remote_memory_test_cpu PASSED in 9.8s //tensorflow/python/eager/polymorphic_function:argument_naming_test_cpu PASSED in 11.1s //tensorflow/python/eager/polymorphic_function:atomic_function_test_cpu PASSED in 14.0s //tensorflow/python/eager/polymorphic_function:collection_test_cpu PASSED in 12.5s //tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu PASSED in 9.8s //tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu_mlir_bridge_test PASSED in 11.6s //tensorflow/python/eager/polymorphic_function:concrete_function_test_cpu PASSED in 13.7s //tensorflow/python/eager/polymorphic_function:function_spec_test PASSED in 10.6s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_jit_test_cpu PASSED in 34.0s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_jit_test_cpu_mlir_bridge_test PASSED in 30.3s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_test_cpu PASSED in 12.4s //tensorflow/python/eager/polymorphic_function:tracing_compilation_test PASSED in 31.1s //tensorflow/python/feature_column:sequence_feature_column_integration_test PASSED in 12.4s //tensorflow/python/feature_column:serialization_test PASSED in 15.9s //tensorflow/python/framework:auto_control_deps_test PASSED in 27.0s //tensorflow/python/framework:c_api_util_test PASSED in 12.3s //tensorflow/python/framework:common_shapes_test PASSED in 8.8s //tensorflow/python/framework:composite_tensor_test PASSED in 30.7s //tensorflow/python/framework:config_test_2gpu PASSED in 14.4s //tensorflow/python/framework:config_test_cpu PASSED in 15.8s //tensorflow/python/framework:constant_op_test PASSED in 12.5s //tensorflow/python/framework:device_spec_test PASSED in 10.0s //tensorflow/python/framework:device_test PASSED in 13.4s //tensorflow/python/framework:dtypes_test PASSED in 53.2s //tensorflow/python/framework:error_interpolation_test PASSED in 10.2s //tensorflow/python/framework:errors_test PASSED in 12.4s //tensorflow/python/framework:extension_type_field_test PASSED in 12.1s //tensorflow/python/framework:extension_type_test PASSED in 52.2s //tensorflow/python/framework:file_system_test PASSED in 11.7s //tensorflow/python/framework:flexible_dtypes_test PASSED in 116.6s //tensorflow/python/framework:function_def_to_graph_test PASSED in 29.2s //tensorflow/python/framework:graph_building_benchmark_cpu PASSED in 10.3s //tensorflow/python/framework:graph_util_test PASSED in 9.2s //tensorflow/python/framework:immutable_dict_test PASSED in 11.9s //tensorflow/python/framework:importer_test PASSED in 11.9s //tensorflow/python/framework:indexed_slices_test PASSED in 9.7s //tensorflow/python/framework:kernels_test PASSED in 20.4s //tensorflow/python/framework:meta_graph_test PASSED in 16.1s //tensorflow/python/framework:node_file_writer_test_cpu PASSED in 9.2s //tensorflow/python/framework:offset_counter_helper_test PASSED in 0.1s //tensorflow/python/framework:op_allowlist_namespace_test PASSED in 5.7s //tensorflow/python/framework:op_callbacks_test_cpu PASSED in 16.5s //tensorflow/python/framework:op_def_library_test PASSED in 10.0s //tensorflow/python/framework:op_def_util_test PASSED in 10.2s //tensorflow/python/framework:ops_enable_eager_test PASSED in 2.8s //tensorflow/python/framework:ops_test PASSED in 23.3s //tensorflow/python/framework:proto_test PASSED in 11.5s //tensorflow/python/framework:py_context_manager_test PASSED in 9.9s //tensorflow/python/framework:python_api_dispatcher_test PASSED in 12.4s //tensorflow/python/framework:python_api_info_test PASSED in 13.5s //tensorflow/python/framework:python_api_parameter_converter_test PASSED in 11.0s //tensorflow/python/framework:python_op_gen_annotation_test PASSED in 5.2s //tensorflow/python/framework:python_op_gen_annotator_test PASSED in 0.4s //tensorflow/python/framework:python_tensor_converter_test PASSED in 11.2s //tensorflow/python/framework:random_seed_test PASSED in 9.9s //tensorflow/python/framework:registry_test PASSED in 10.9s //tensorflow/python/framework:smart_cond_test PASSED in 14.1s //tensorflow/python/framework:sparse_tensor_test PASSED in 25.9s //tensorflow/python/framework:subscribe_test PASSED in 32.8s //tensorflow/python/framework:tensor_shape_test PASSED in 12.4s //tensorflow/python/framework:tensor_test PASSED in 18.9s //tensorflow/python/framework:tensor_util_test PASSED in 15.7s //tensorflow/python/framework:test_combinations_test PASSED in 17.3s //tensorflow/python/framework:test_util_test_cpu PASSED in 29.2s //tensorflow/python/framework:tf2_test PASSED in 17.4s //tensorflow/python/framework:traceable_stack_test PASSED in 10.2s //tensorflow/python/framework:type_spec_test PASSED in 10.6s //tensorflow/python/framework:versions_test PASSED in 12.6s //tensorflow/python/framework:weak_tensor_test PASSED in 15.0s //tensorflow/python/framework/experimental:graph_building_test_cpu PASSED in 12.6s //tensorflow/python/framework/experimental:unified_api_test_cpu PASSED in 16.6s //tensorflow/python/grappler:arithmetic_optimizer_test_cpu PASSED in 12.7s //tensorflow/python/grappler:auto_mixed_precision_test_cpu PASSED in 20.2s //tensorflow/python/grappler:constant_folding_test_cpu PASSED in 11.3s //tensorflow/python/grappler:cost_analyzer_test PASSED in 25.2s //tensorflow/python/grappler:datasets_test PASSED in 12.8s //tensorflow/python/grappler:item_test PASSED in 28.0s //tensorflow/python/grappler:memory_optimizer_test PASSED in 18.7s //tensorflow/python/grappler:model_analyzer_test PASSED in 10.3s //tensorflow/python/grappler:remapper_test_cpu PASSED in 16.0s //tensorflow/python/grappler:tf_optimizer_test PASSED in 12.7s //tensorflow/python/kernel_tests:benchmark_test_cpu PASSED in 16.0s //tensorflow/python/kernel_tests:check_ops_test_cpu PASSED in 19.8s //tensorflow/python/kernel_tests:collective_ops_multi_worker_test PASSED in 47.3s //tensorflow/python/kernel_tests:composite_tensor_ops_test PASSED in 11.1s //tensorflow/python/kernel_tests:critical_section_test_cpu PASSED in 22.1s //tensorflow/python/kernel_tests:garbage_collection_test PASSED in 10.9s //tensorflow/python/kernel_tests:gradient_correctness_test_cpu PASSED in 17.2s //tensorflow/python/kernel_tests:histogram_ops_test_cpu PASSED in 11.8s //tensorflow/python/kernel_tests:logging_ops_test_cpu PASSED in 12.1s //tensorflow/python/kernel_tests:numerics_test_cpu PASSED in 12.9s //tensorflow/python/kernel_tests:template_test PASSED in 12.3s //tensorflow/python/kernel_tests:trace_op_test_cpu PASSED in 10.2s //tensorflow/python/kernel_tests/array_ops:batch_gather_op_test_cpu PASSED in 20.2s //tensorflow/python/kernel_tests/array_ops:batch_scatter_ops_test PASSED in 19.4s //tensorflow/python/kernel_tests/array_ops:batchtospace_op_test_cpu PASSED in 12.1s //tensorflow/python/kernel_tests/array_ops:bcast_ops_test PASSED in 19.8s //tensorflow/python/kernel_tests/array_ops:bitcast_op_test_cpu PASSED in 10.6s //tensorflow/python/kernel_tests/array_ops:broadcast_to_ops_test_cpu PASSED in 38.4s //tensorflow/python/kernel_tests/array_ops:cast_op_test_cpu PASSED in 42.3s //tensorflow/python/kernel_tests/array_ops:constant_op_eager_test_cpu PASSED in 11.6s //tensorflow/python/kernel_tests/array_ops:constant_op_test_cpu PASSED in 12.5s //tensorflow/python/kernel_tests/array_ops:denormal_test_cpu PASSED in 9.9s //tensorflow/python/kernel_tests/array_ops:depthtospace_op_test_cpu PASSED in 34.8s //tensorflow/python/kernel_tests/array_ops:edit_distance_op_test PASSED in 9.8s //tensorflow/python/kernel_tests/array_ops:fingerprint_op_test PASSED in 13.9s //tensorflow/python/kernel_tests/array_ops:gather_nd_op_test_cpu PASSED in 10.3s //tensorflow/python/kernel_tests/array_ops:identity_n_op_py_test PASSED in 11.6s //tensorflow/python/kernel_tests/array_ops:identity_op_py_test PASSED in 16.1s //tensorflow/python/kernel_tests/array_ops:large_concat_op_test_cpu PASSED in 18.6s //tensorflow/python/kernel_tests/array_ops:manip_ops_test_cpu PASSED in 13.1s //tensorflow/python/kernel_tests/array_ops:one_hot_op_test_cpu PASSED in 12.3s //tensorflow/python/kernel_tests/array_ops:pad_op_test_cpu PASSED in 18.7s //tensorflow/python/kernel_tests/array_ops:reshape_op_test_cpu PASSED in 12.4s //tensorflow/python/kernel_tests/array_ops:reverse_sequence_op_test_cpu PASSED in 10.2s //tensorflow/python/kernel_tests/array_ops:scalar_test_cpu PASSED in 12.4s //tensorflow/python/kernel_tests/array_ops:shape_ops_test_cpu PASSED in 18.0s //tensorflow/python/kernel_tests/array_ops:slice_op_test_cpu PASSED in 12.7s //tensorflow/python/kernel_tests/array_ops:spacetobatch_op_test_cpu PASSED in 16.7s //tensorflow/python/kernel_tests/array_ops:spacetodepth_op_test_cpu PASSED in 13.8s //tensorflow/python/kernel_tests/array_ops:stack_op_test_cpu PASSED in 20.2s //tensorflow/python/kernel_tests/array_ops:unique_op_test_cpu PASSED in 10.5s //tensorflow/python/kernel_tests/array_ops:unstack_op_test_cpu PASSED in 14.0s //tensorflow/python/kernel_tests/array_ops:where_op_test_cpu PASSED in 16.8s //tensorflow/python/kernel_tests/control_flow:cond_v2_test_cpu PASSED in 77.0s //tensorflow/python/kernel_tests/control_flow:control_flow_util_test PASSED in 13.1s //tensorflow/python/kernel_tests/control_flow:control_flow_util_v2_test PASSED in 14.6s //tensorflow/python/kernel_tests/control_flow:py_func_test_cpu PASSED in 23.5s //tensorflow/python/kernel_tests/control_flow:scan_ops_test_cpu PASSED in 53.0s //tensorflow/python/kernel_tests/control_flow:while_v2_test_cpu PASSED in 91.2s //tensorflow/python/kernel_tests/custom_ops:ackermann_test PASSED in 13.2s //tensorflow/python/kernel_tests/custom_ops:duplicate_op_test PASSED in 8.5s //tensorflow/python/kernel_tests/custom_ops:invalid_op_test PASSED in 16.1s //tensorflow/python/kernel_tests/data_structures:conditional_accumulator_test PASSED in 12.5s //tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_2gpu PASSED in 15.0s //tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_cpu PASSED in 17.4s //tensorflow/python/kernel_tests/data_structures:dynamic_stitch_op_test_cpu PASSED in 10.3s //tensorflow/python/kernel_tests/data_structures:fifo_queue_test PASSED in 14.6s //tensorflow/python/kernel_tests/data_structures:list_ops_test_cpu PASSED in 43.0s //tensorflow/python/kernel_tests/data_structures:listdiff_op_test PASSED in 9.9s //tensorflow/python/kernel_tests/data_structures:lookup_ops_test PASSED in 32.3s //tensorflow/python/kernel_tests/data_structures:map_ops_test PASSED in 22.0s //tensorflow/python/kernel_tests/data_structures:padding_fifo_queue_test_cpu PASSED in 10.4s //tensorflow/python/kernel_tests/data_structures:priority_queue_test PASSED in 13.8s //tensorflow/python/kernel_tests/data_structures:stack_ops_test_cpu PASSED in 9.8s //tensorflow/python/kernel_tests/data_structures:stage_op_test_cpu PASSED in 11.1s //tensorflow/python/kernel_tests/distributions:bernoulli_test_cpu PASSED in 20.8s //tensorflow/python/kernel_tests/distributions:bijector_test_cpu PASSED in 16.1s //tensorflow/python/kernel_tests/distributions:categorical_test_cpu PASSED in 18.3s //tensorflow/python/kernel_tests/distributions:dirichlet_multinomial_test_cpu PASSED in 20.4s //tensorflow/python/kernel_tests/distributions:dirichlet_test_cpu PASSED in 23.5s //tensorflow/python/kernel_tests/distributions:exponential_test_cpu PASSED in 18.0s //tensorflow/python/kernel_tests/distributions:gamma_test_cpu PASSED in 56.5s //tensorflow/python/kernel_tests/distributions:identity_bijector_test_cpu PASSED in 10.1s //tensorflow/python/kernel_tests/distributions:kullback_leibler_test_cpu PASSED in 14.1s //tensorflow/python/kernel_tests/distributions:laplace_test_cpu PASSED in 42.4s //tensorflow/python/kernel_tests/distributions:multinomial_test_cpu PASSED in 10.0s //tensorflow/python/kernel_tests/distributions:normal_test_cpu PASSED in 44.9s //tensorflow/python/kernel_tests/distributions:special_math_test_cpu PASSED in 25.4s //tensorflow/python/kernel_tests/distributions:uniform_test_cpu PASSED in 15.1s //tensorflow/python/kernel_tests/image_ops:attention_ops_test PASSED in 14.6s //tensorflow/python/kernel_tests/image_ops:decode_bmp_op_test PASSED in 11.1s //tensorflow/python/kernel_tests/image_ops:decode_compressed_op_test PASSED in 11.5s //tensorflow/python/kernel_tests/image_ops:decode_image_op_test PASSED in 14.5s //tensorflow/python/kernel_tests/image_ops:decode_jpeg_op_test PASSED in 10.9s //tensorflow/python/kernel_tests/image_ops:decode_png_op_test PASSED in 13.6s //tensorflow/python/kernel_tests/image_ops:decode_raw_op_test PASSED in 17.0s //tensorflow/python/kernel_tests/image_ops:draw_bounding_box_op_test_cpu PASSED in 30.3s //tensorflow/python/kernel_tests/image_ops:extract_image_patches_op_test_cpu PASSED in 10.2s //tensorflow/python/kernel_tests/image_ops:extract_volume_patches_op_test_cpu PASSED in 9.8s //tensorflow/python/kernel_tests/io_ops:checkpoint_ops_test PASSED in 11.4s //tensorflow/python/kernel_tests/io_ops:decode_csv_op_test PASSED in 27.2s //tensorflow/python/kernel_tests/io_ops:io_ops_test PASSED in 9.8s //tensorflow/python/kernel_tests/io_ops:parse_single_example_op_test PASSED in 11.3s //tensorflow/python/kernel_tests/io_ops:parsing_ops_test PASSED in 39.5s //tensorflow/python/kernel_tests/io_ops:reader_ops_test PASSED in 13.3s //tensorflow/python/kernel_tests/io_ops:record_input_test PASSED in 33.2s //tensorflow/python/kernel_tests/io_ops:save_restore_ops_test PASSED in 28.1s //tensorflow/python/kernel_tests/linalg:determinant_op_test_cpu PASSED in 11.6s //tensorflow/python/kernel_tests/linalg:linear_operator_addition_test_cpu PASSED in 11.5s //tensorflow/python/kernel_tests/linalg:linear_operator_algebra_test_cpu PASSED in 12.3s //tensorflow/python/kernel_tests/linalg:linear_operator_test_cpu PASSED in 12.1s //tensorflow/python/kernel_tests/linalg:lu_op_test_cpu PASSED in 11.8s //tensorflow/python/kernel_tests/linalg:matrix_inverse_op_test_cpu PASSED in 14.8s //tensorflow/python/kernel_tests/linalg:matrix_logarithm_op_test PASSED in 75.6s //tensorflow/python/kernel_tests/linalg:matrix_solve_ls_op_test_cpu PASSED in 42.8s //tensorflow/python/kernel_tests/linalg:matrix_solve_op_test_cpu PASSED in 38.8s //tensorflow/python/kernel_tests/linalg:matrix_square_root_op_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/linalg:slicing_test_cpu PASSED in 15.4s //tensorflow/python/kernel_tests/linalg/sparse:conjugate_gradient_test_cpu PASSED in 17.2s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_test_cpu PASSED in 13.0s //tensorflow/python/kernel_tests/math_ops:aggregate_ops_test_cpu PASSED in 18.8s //tensorflow/python/kernel_tests/math_ops:argmax_op_test_cpu PASSED in 19.5s //tensorflow/python/kernel_tests/math_ops:banded_triangular_solve_op_test_cpu PASSED in 12.4s //tensorflow/python/kernel_tests/math_ops:basic_gpu_test_cpu PASSED in 10.6s //tensorflow/python/kernel_tests/math_ops:bincount_op_test_cpu PASSED in 11.3s //tensorflow/python/kernel_tests/math_ops:bucketize_op_test_cpu PASSED in 14.1s //tensorflow/python/kernel_tests/math_ops:clip_ops_test PASSED in 13.5s //tensorflow/python/kernel_tests/math_ops:confusion_matrix_test PASSED in 13.5s //tensorflow/python/kernel_tests/math_ops:cross_grad_test_cpu PASSED in 10.1s //tensorflow/python/kernel_tests/math_ops:cumulative_logsumexp_test_cpu PASSED in 18.3s //tensorflow/python/kernel_tests/math_ops:in_topk_op_test_cpu PASSED in 15.2s //tensorflow/python/kernel_tests/math_ops:reduce_benchmark_test_cpu PASSED in 10.1s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_d9m_test_cpu PASSED in 9.7s //tensorflow/python/kernel_tests/math_ops:sets_test PASSED in 28.1s //tensorflow/python/kernel_tests/math_ops:topk_op_test_cpu PASSED in 10.5s //tensorflow/python/kernel_tests/math_ops:zero_division_test_cpu PASSED in 11.8s //tensorflow/python/kernel_tests/nn_ops:betainc_op_test_cpu PASSED in 14.2s //tensorflow/python/kernel_tests/nn_ops:bias_op_test_cpu PASSED in 151.9s //tensorflow/python/kernel_tests/nn_ops:conv1d_test_cpu PASSED in 35.8s //tensorflow/python/kernel_tests/nn_ops:conv1d_transpose_test_cpu PASSED in 12.5s //tensorflow/python/kernel_tests/nn_ops:conv2d_transpose_test_cpu PASSED in 16.3s //tensorflow/python/kernel_tests/nn_ops:conv3d_backprop_filter_v2_grad_test_cpu PASSED in 13.8s //tensorflow/python/kernel_tests/nn_ops:conv3d_transpose_test_cpu PASSED in 12.7s //tensorflow/python/kernel_tests/nn_ops:ctc_decoder_ops_test PASSED in 10.5s //tensorflow/python/kernel_tests/nn_ops:ctc_loss_op_test_cpu PASSED in 62.7s //tensorflow/python/kernel_tests/nn_ops:cudnn_d9m_test_cpu PASSED in 20.1s //tensorflow/python/kernel_tests/nn_ops:cudnn_deterministic_ops_test_cpu PASSED in 11.3s //tensorflow/python/kernel_tests/nn_ops:losses_test PASSED in 37.7s //tensorflow/python/kernel_tests/nn_ops:lrn_op_test_cpu PASSED in 30.2s //tensorflow/python/kernel_tests/nn_ops:morphological_ops_test_cpu PASSED in 14.1s //tensorflow/python/kernel_tests/nn_ops:nth_element_op_test_cpu PASSED in 12.5s //tensorflow/python/kernel_tests/nn_ops:pool_test_cpu PASSED in 29.3s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_3d_test_cpu PASSED in 53.2s //tensorflow/python/kernel_tests/nn_ops:relu_op_test_cpu PASSED in 12.2s //tensorflow/python/kernel_tests/nn_ops:softmax_op_test_cpu PASSED in 11.2s //tensorflow/python/kernel_tests/nn_ops:softplus_op_test_cpu PASSED in 11.1s //tensorflow/python/kernel_tests/nn_ops:softsign_op_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/nn_ops:xent_op_d9m_test_cpu PASSED in 123.2s //tensorflow/python/kernel_tests/nn_ops:xent_op_test_cpu PASSED in 13.2s //tensorflow/python/kernel_tests/proto:descriptor_source_test PASSED in 11.6s //tensorflow/python/kernel_tests/proto:encode_proto_op_test PASSED in 11.1s //tensorflow/python/kernel_tests/quantization_ops:quantization_ops_test PASSED in 12.1s //tensorflow/python/kernel_tests/random:candidate_sampler_ops_test PASSED in 10.5s //tensorflow/python/kernel_tests/random:multinomial_op_test_cpu PASSED in 36.4s //tensorflow/python/kernel_tests/random:parameterized_truncated_normal_op_test_cpu PASSED in 26.7s //tensorflow/python/kernel_tests/random:random_crop_test_cpu PASSED in 12.1s //tensorflow/python/kernel_tests/random:random_grad_test_cpu PASSED in 50.2s //tensorflow/python/kernel_tests/random:random_ops_test_cpu PASSED in 16.3s //tensorflow/python/kernel_tests/random:random_poisson_test_cpu PASSED in 34.3s //tensorflow/python/kernel_tests/random:random_shuffle_queue_test PASSED in 10.0s //tensorflow/python/kernel_tests/random:stateful_random_ops_test_cpu PASSED in 24.3s //tensorflow/python/kernel_tests/signal:mel_ops_test_cpu PASSED in 19.9s //tensorflow/python/kernel_tests/signal:mfcc_ops_test_cpu PASSED in 10.0s //tensorflow/python/kernel_tests/signal:reconstruction_ops_test_cpu PASSED in 14.6s //tensorflow/python/kernel_tests/signal:shape_ops_test_cpu PASSED in 24.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_add_op_test PASSED in 18.8s //tensorflow/python/kernel_tests/sparse_ops:sparse_concat_op_test PASSED in 13.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_conditional_accumulator_test PASSED in 10.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_cross_op_test PASSED in 18.2s //tensorflow/python/kernel_tests/sparse_ops:sparse_matmul_op_test_cpu PASSED in 38.4s //tensorflow/python/kernel_tests/sparse_ops:sparse_reorder_op_test PASSED in 10.5s //tensorflow/python/kernel_tests/sparse_ops:sparse_reshape_op_test PASSED in 11.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_serialization_ops_test PASSED in 12.2s //tensorflow/python/kernel_tests/sparse_ops:sparse_slice_op_test PASSED in 9.4s //tensorflow/python/kernel_tests/sparse_ops:sparse_split_op_test_cpu PASSED in 28.4s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_grad_test_cpu PASSED in 19.7s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_d9m_test_cpu PASSED in 39.4s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_test_cpu PASSED in 29.8s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensors_map_ops_test PASSED in 12.5s //tensorflow/python/kernel_tests/sparse_ops:sparse_to_dense_op_py_test_cpu PASSED in 36.5s //tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_d9m_test_cpu PASSED in 77.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_test_cpu PASSED in 13.2s //tensorflow/python/kernel_tests/sparse_ops:sparsemask_op_test PASSED in 10.3s //tensorflow/python/kernel_tests/strings_ops:as_string_op_test PASSED in 11.7s //tensorflow/python/kernel_tests/strings_ops:base64_ops_test PASSED in 11.0s //tensorflow/python/kernel_tests/strings_ops:reduce_join_op_test_cpu PASSED in 12.3s //tensorflow/python/kernel_tests/strings_ops:regex_full_match_op_test PASSED in 13.6s //tensorflow/python/kernel_tests/strings_ops:regex_replace_op_test PASSED in 12.3s //tensorflow/python/kernel_tests/strings_ops:string_bytes_split_op_test PASSED in 10.9s //tensorflow/python/kernel_tests/strings_ops:string_format_op_test PASSED in 12.8s //tensorflow/python/kernel_tests/strings_ops:string_join_op_test PASSED in 9.8s //tensorflow/python/kernel_tests/strings_ops:string_length_op_test PASSED in 11.3s //tensorflow/python/kernel_tests/strings_ops:string_lower_op_test PASSED in 11.4s //tensorflow/python/kernel_tests/strings_ops:string_split_op_test PASSED in 13.0s //tensorflow/python/kernel_tests/strings_ops:string_strip_op_test PASSED in 10.5s //tensorflow/python/kernel_tests/strings_ops:string_to_hash_bucket_op_test_cpu PASSED in 27.0s //tensorflow/python/kernel_tests/strings_ops:string_to_number_op_test_cpu PASSED in 11.2s //tensorflow/python/kernel_tests/strings_ops:string_upper_op_test PASSED in 9.4s //tensorflow/python/kernel_tests/strings_ops:substr_op_test PASSED in 34.1s //tensorflow/python/kernel_tests/strings_ops:unicode_decode_op_test PASSED in 30.5s //tensorflow/python/kernel_tests/strings_ops:unicode_encode_op_test PASSED in 13.6s //tensorflow/python/kernel_tests/strings_ops:unicode_script_op_test PASSED in 10.0s //tensorflow/python/kernel_tests/strings_ops:unicode_transcode_op_test PASSED in 10.1s //tensorflow/python/kernel_tests/strings_ops:unsorted_segment_join_op_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/summary_ops:summary_ops_test_cpu PASSED in 18.3s //tensorflow/python/kernel_tests/summary_ops:summary_v1_audio_op_test_cpu PASSED in 24.6s //tensorflow/python/kernel_tests/summary_ops:summary_v1_image_op_test_cpu PASSED in 11.2s //tensorflow/python/kernel_tests/summary_ops:summary_v1_ops_test PASSED in 10.9s //tensorflow/python/kernel_tests/summary_ops:summary_v1_tensor_op_test PASSED in 10.0s //tensorflow/python/kernel_tests/v1_compat_tests:array_ops_test_cpu PASSED in 12.3s //tensorflow/python/kernel_tests/v1_compat_tests:dense_update_ops_test_cpu PASSED in 11.1s //tensorflow/python/kernel_tests/v1_compat_tests:identity_op_py_test PASSED in 11.3s //tensorflow/python/kernel_tests/v1_compat_tests:scatter_nd_ops_test_cpu PASSED in 12.0s //tensorflow/python/kernel_tests/v1_compat_tests:session_ops_test_cpu PASSED in 12.8s //tensorflow/python/kernel_tests/v1_compat_tests:stack_op_test_cpu PASSED in 9.2s //tensorflow/python/kernel_tests/variables:dense_update_ops_no_tsan_test_cpu PASSED in 15.6s //tensorflow/python/kernel_tests/variables:dense_update_ops_test_cpu PASSED in 10.7s //tensorflow/python/kernel_tests/variables:partitioned_variables_test PASSED in 13.3s //tensorflow/python/kernel_tests/variables:resource_variable_ops_test_cpu PASSED in 50.3s //tensorflow/python/kernel_tests/variables:variable_ops_test_cpu PASSED in 11.4s //tensorflow/python/kernel_tests/variables:variable_scope_test PASSED in 39.7s //tensorflow/python/kernel_tests/variables:variables_test PASSED in 10.5s //tensorflow/python/lib/io:file_io_test PASSED in 15.6s //tensorflow/python/lib/io:tf_record_test PASSED in 13.1s //tensorflow/python/module:module_test PASSED in 10.6s //tensorflow/python/ops:array_grad_test_cpu PASSED in 11.7s //tensorflow/python/ops:array_ops_shape_test PASSED in 8.4s //tensorflow/python/ops:array_ops_test PASSED in 10.1s //tensorflow/python/ops:autograph_ops_test PASSED in 9.7s //tensorflow/python/ops:batch_norm_benchmark_cpu PASSED in 26.1s //tensorflow/python/ops:bincount_ops_test_cpu PASSED in 10.8s //tensorflow/python/ops:bitwise_ops_test_cpu PASSED in 14.4s //tensorflow/python/ops:clip_ops_test PASSED in 11.3s //tensorflow/python/ops:clustering_ops_test PASSED in 25.1s //tensorflow/python/ops:collective_ops_benchmark_cpu PASSED in 10.7s //tensorflow/python/ops:collective_ops_gpu_test_2gpu PASSED in 14.0s //tensorflow/python/ops:collective_ops_gpu_test_cpu PASSED in 11.1s //tensorflow/python/ops:collective_ops_test PASSED in 36.6s //tensorflow/python/ops:collective_ops_xla_test PASSED in 9.7s //tensorflow/python/ops:compiled_collective_ops_gpu_test_2gpu PASSED in 13.5s //tensorflow/python/ops:compiled_collective_ops_gpu_test_cpu PASSED in 13.3s //tensorflow/python/ops:concat_benchmark_cpu PASSED in 10.1s //tensorflow/python/ops:control_flow_ops_benchmark_cpu PASSED in 9.2s //tensorflow/python/ops:control_flow_v2_enable_test PASSED in 12.2s //tensorflow/python/ops:control_flow_v2_toggles_test PASSED in 14.5s //tensorflow/python/ops:dequantize_op_test PASSED in 10.5s //tensorflow/python/ops:embedding_ops_test_cpu PASSED in 12.5s //tensorflow/python/ops:factory_ops_test_cpu PASSED in 18.3s //tensorflow/python/ops:functional_ops_test PASSED in 10.2s //tensorflow/python/ops:gradient_checker_v2_test_cpu PASSED in 25.9s //tensorflow/python/ops:gradients_test_cpu PASSED in 38.3s //tensorflow/python/ops:init_ops_test_cpu PASSED in 30.7s //tensorflow/python/ops:init_ops_v2_test_cpu PASSED in 14.6s //tensorflow/python/ops:math_grad_test_cpu PASSED in 21.8s //tensorflow/python/ops:math_ops_linspace_test_cpu PASSED in 14.5s //tensorflow/python/ops:math_ops_test_cpu PASSED in 33.5s //tensorflow/python/ops:matmul_benchmark_cpu PASSED in 10.4s //tensorflow/python/ops:nn_grad_test_cpu PASSED in 11.7s //tensorflow/python/ops:nn_loss_scaling_utilities_test PASSED in 13.8s //tensorflow/python/ops:nn_test_cpu PASSED in 97.7s //tensorflow/python/ops:nn_xent_test_cpu PASSED in 11.5s //tensorflow/python/ops:op_selector_test PASSED in 27.4s //tensorflow/python/ops:quantized_conv_ops_test PASSED in 10.9s //tensorflow/python/ops:quantized_ops_test PASSED in 10.4s //tensorflow/python/ops:raw_ops_test_cpu PASSED in 10.3s //tensorflow/python/ops:rnn_grad_test_cpu PASSED in 10.7s //tensorflow/python/ops:script_ops_test PASSED in 9.9s //tensorflow/python/ops:sort_ops_test PASSED in 10.5s //tensorflow/python/ops:sparse_bincount_ops_test_cpu PASSED in 14.6s //tensorflow/python/ops:sparse_ops_test PASSED in 21.7s //tensorflow/python/ops:split_benchmark_cpu PASSED in 10.2s //tensorflow/python/ops:tensor_array_ops_test PASSED in 9.5s //tensorflow/python/ops:transpose_benchmark_cpu PASSED in 9.3s //tensorflow/python/ops:variable_spec_test PASSED in 19.1s //tensorflow/python/ops:weak_tensor_array_ops_test PASSED in 10.4s //tensorflow/python/ops:weak_tensor_image_ops_test PASSED in 9.1s //tensorflow/python/ops:weak_tensor_math_ops_test PASSED in 14.2s //tensorflow/python/ops:weak_tensor_nn_test_cpu PASSED in 18.7s //tensorflow/python/ops:weak_tensor_np_array_ops_test PASSED in 38.7s //tensorflow/python/ops:weak_tensor_np_math_ops_test PASSED in 12.1s //tensorflow/python/ops:weak_tensor_ops_test PASSED in 15.0s //tensorflow/python/ops/losses:util_test PASSED in 13.5s //tensorflow/python/ops/memory_tests:custom_gradient_memory_test_cpu PASSED in 15.7s //tensorflow/python/ops/numpy_ops:np_array_ops_test_cpu PASSED in 117.0s //tensorflow/python/ops/numpy_ops:np_arrays_test_cpu PASSED in 10.6s //tensorflow/python/ops/numpy_ops:np_dtypes_test_cpu PASSED in 13.8s //tensorflow/python/ops/numpy_ops:np_interop_test_cpu PASSED in 65.1s //tensorflow/python/ops/numpy_ops:np_logic_test_cpu PASSED in 16.7s //tensorflow/python/ops/numpy_ops:np_math_ops_test_cpu PASSED in 31.9s //tensorflow/python/ops/numpy_ops:np_random_test_cpu PASSED in 71.9s //tensorflow/python/ops/numpy_ops:np_utils_test_cpu PASSED in 11.2s //tensorflow/python/ops/numpy_ops/integration_test:np_config_test_cpu PASSED in 23.8s //tensorflow/python/ops/numpy_ops/integration_test:public_symbol_test PASSED in 36.5s //tensorflow/python/ops/parallel_for:array_test_cpu PASSED in 37.1s //tensorflow/python/ops/parallel_for:gradients_test_cpu PASSED in 21.2s //tensorflow/python/ops/parallel_for:xla_control_flow_ops_test_cpu PASSED in 61.9s //tensorflow/python/ops/ragged:convert_to_tensor_or_ragged_tensor_op_test PASSED in 9.9s //tensorflow/python/ops/ragged:ragged_batch_gather_op_test PASSED in 50.8s //tensorflow/python/ops/ragged:ragged_bincount_ops_test_cpu PASSED in 30.5s //tensorflow/python/ops/ragged:ragged_bitcast_op_test PASSED in 25.8s //tensorflow/python/ops/ragged:ragged_boolean_mask_op_test PASSED in 22.1s //tensorflow/python/ops/ragged:ragged_concat_op_test PASSED in 29.4s //tensorflow/python/ops/ragged:ragged_const_op_test PASSED in 10.3s //tensorflow/python/ops/ragged:ragged_constant_value_op_test PASSED in 25.7s //tensorflow/python/ops/ragged:ragged_cross_op_test PASSED in 28.9s //tensorflow/python/ops/ragged:ragged_dispatch_test PASSED in 154.2s //tensorflow/python/ops/ragged:ragged_dynamic_partition_op_test_cpu PASSED in 19.0s //tensorflow/python/ops/ragged:ragged_eager_test PASSED in 10.6s //tensorflow/python/ops/ragged:ragged_expand_dims_op_test PASSED in 10.6s //tensorflow/python/ops/ragged:ragged_factory_ops_test_cpu PASSED in 27.8s //tensorflow/python/ops/ragged:ragged_fill_empty_rows_op_test PASSED in 12.2s //tensorflow/python/ops/ragged:ragged_from_sparse_op_test PASSED in 25.7s //tensorflow/python/ops/ragged:ragged_from_tensor_op_test PASSED in 27.6s //tensorflow/python/ops/ragged:ragged_gather_nd_op_test PASSED in 13.6s //tensorflow/python/ops/ragged:ragged_map_flat_values_op_test PASSED in 14.4s //tensorflow/python/ops/ragged:ragged_map_fn_op_test PASSED in 29.6s //tensorflow/python/ops/ragged:ragged_math_ops_test PASSED in 16.8s //tensorflow/python/ops/ragged:ragged_matmul_op_test PASSED in 47.2s //tensorflow/python/ops/ragged:ragged_merge_dims_op_test PASSED in 39.3s //tensorflow/python/ops/ragged:ragged_one_hot_op_test PASSED in 12.9s //tensorflow/python/ops/ragged:ragged_operators_test PASSED in 26.8s //tensorflow/python/ops/ragged:ragged_placeholder_op_test PASSED in 10.5s //tensorflow/python/ops/ragged:ragged_print_op_test PASSED in 16.2s //tensorflow/python/ops/ragged:ragged_range_op_test PASSED in 10.8s //tensorflow/python/ops/ragged:ragged_rank_op_test PASSED in 10.2s //tensorflow/python/ops/ragged:ragged_reduce_op_test PASSED in 36.4s //tensorflow/python/ops/ragged:ragged_resize_image_op_test PASSED in 43.9s //tensorflow/python/ops/ragged:ragged_reverse_op_test PASSED in 20.4s //tensorflow/python/ops/ragged:ragged_row_lengths_op_test PASSED in 27.2s //tensorflow/python/ops/ragged:ragged_row_splits_to_segment_ids_op_test PASSED in 10.0s //tensorflow/python/ops/ragged:ragged_segment_ids_to_row_splits_op_test PASSED in 10.2s //tensorflow/python/ops/ragged:ragged_segment_op_test PASSED in 14.5s //tensorflow/python/ops/ragged:ragged_size_op_test PASSED in 8.6s //tensorflow/python/ops/ragged:ragged_split_op_test PASSED in 41.9s //tensorflow/python/ops/ragged:ragged_squeeze_op_test PASSED in 22.9s //tensorflow/python/ops/ragged:ragged_stack_op_test PASSED in 15.4s //tensorflow/python/ops/ragged:ragged_tensor_bounding_shape_op_test PASSED in 19.3s //tensorflow/python/ops/ragged:ragged_tensor_shape_test PASSED in 77.5s //tensorflow/python/ops/ragged:ragged_tile_op_test PASSED in 55.9s //tensorflow/python/ops/ragged:ragged_to_sparse_op_test PASSED in 11.1s //tensorflow/python/ops/ragged:ragged_to_tensor_op_test PASSED in 85.2s //tensorflow/python/ops/ragged:ragged_util_test PASSED in 43.6s //tensorflow/python/ops/ragged:ragged_where_op_test PASSED in 35.6s //tensorflow/python/ops/ragged:row_partition_test PASSED in 32.2s //tensorflow/python/ops/ragged:string_ngrams_op_test PASSED in 10.7s //tensorflow/python/ops/ragged:strings_reduce_join_op_test PASSED in 13.5s //tensorflow/python/ops/structured:structured_array_ops_test PASSED in 60.8s //tensorflow/python/ops/structured:structured_tensor_slice_test PASSED in 63.5s //tensorflow/python/ops/structured:structured_tensor_spec_test PASSED in 12.7s //tensorflow/python/ops/structured:structured_tensor_test PASSED in 51.7s //tensorflow/python/ops/v1_compat_tests:gradient_checker_test_cpu PASSED in 17.6s //tensorflow/python/platform:benchmark_test PASSED in 11.5s //tensorflow/python/platform:build_info_test PASSED in 17.5s //tensorflow/python/platform:resource_loader_test PASSED in 4.3s //tensorflow/python/profiler:pprof_profiler_test PASSED in 9.2s //tensorflow/python/profiler:profile_context_test_cpu PASSED in 34.0s //tensorflow/python/profiler:profiler_client_test_cpu PASSED in 10.4s //tensorflow/python/profiler:profiler_test_cpu PASSED in 22.7s //tensorflow/python/profiler:profiler_v2_test_cpu PASSED in 10.8s //tensorflow/python/profiler:profiler_wrapper_test PASSED in 8.9s //tensorflow/python/profiler:tfprof_logger_test PASSED in 10.0s //tensorflow/python/profiler/internal:flops_registry_test PASSED in 9.9s //tensorflow/python/profiler/internal:print_model_analysis_test PASSED in 10.0s //tensorflow/python/profiler/internal:run_metadata_test_cpu PASSED in 23.4s //tensorflow/python/saved_model:fingerprinting_test PASSED in 14.8s //tensorflow/python/saved_model:keras_injection_test PASSED in 29.6s //tensorflow/python/saved_model:load_v1_in_v2_test PASSED in 20.8s //tensorflow/python/saved_model:loader_test PASSED in 17.5s //tensorflow/python/saved_model:method_name_updater_test PASSED in 9.3s //tensorflow/python/saved_model:metrics_test PASSED in 15.0s //tensorflow/python/saved_model:nested_structure_coder_test PASSED in 12.4s //tensorflow/python/saved_model:pywrap_saved_model_fingerprinting_test PASSED in 9.7s //tensorflow/python/saved_model:pywrap_saved_model_metrics_test PASSED in 10.7s //tensorflow/python/saved_model:revived_types_test PASSED in 10.2s //tensorflow/python/saved_model:save_context_test PASSED in 11.5s //tensorflow/python/saved_model:save_test PASSED in 36.8s //tensorflow/python/saved_model:saved_model_test PASSED in 30.0s //tensorflow/python/saved_model:signature_def_utils_test PASSED in 13.6s //tensorflow/python/saved_model:simple_save_test PASSED in 9.8s //tensorflow/python/saved_model:tracing_utils_test PASSED in 9.9s //tensorflow/python/saved_model:utils_test PASSED in 9.8s //tensorflow/python/saved_model/model_utils:export_output_test PASSED in 10.3s //tensorflow/python/saved_model/model_utils:export_test PASSED in 15.4s //tensorflow/python/saved_model/model_utils:mode_keys_test PASSED in 9.0s //tensorflow/python/saved_model/registration:registration_saving_test PASSED in 25.8s //tensorflow/python/saved_model/registration:registration_test PASSED in 11.6s //tensorflow/python/saved_model/registration:tf_registration_test PASSED in 26.6s //tensorflow/python/saved_model/tests:variable_wrapper_test PASSED in 10.1s //tensorflow/python/summary:plugin_asset_test PASSED in 9.3s //tensorflow/python/summary:summary_iterator_test PASSED in 9.9s //tensorflow/python/summary:summary_test PASSED in 13.3s //tensorflow/python/summary:summary_v2_test PASSED in 18.9s //tensorflow/python/summary/writer:writer_test PASSED in 21.1s //tensorflow/python/tools:aot_compiled_test PASSED in 23.9s //tensorflow/python/tools:freeze_graph_test PASSED in 9.3s //tensorflow/python/tools:optimize_for_inference_test PASSED in 10.4s //tensorflow/python/tools:print_selective_registration_header_test PASSED in 39.4s //tensorflow/python/tools:saved_model_cli_test PASSED in 58.4s //tensorflow/python/tools:saved_model_utils_test PASSED in 12.4s //tensorflow/python/tools:strip_unused_test PASSED in 11.8s //tensorflow/python/tools/api/generator:create_python_api_test PASSED in 12.4s //tensorflow/python/tools/api/generator:output_init_files_test PASSED in 18.8s //tensorflow/python/tools/api/generator:tensorflow_doc_srcs_test PASSED in 14.2s //tensorflow/python/tpu:bfloat16_test PASSED in 9.4s //tensorflow/python/tpu:feature_column_test PASSED in 14.3s //tensorflow/python/tpu:topology_test PASSED in 11.7s //tensorflow/python/tpu:tpu_embedding_for_serving_test PASSED in 11.7s //tensorflow/python/tpu:tpu_embedding_v2_utils_test PASSED in 9.3s //tensorflow/python/tpu:tpu_infeed_test PASSED in 10.0s //tensorflow/python/tpu:tpu_sharding_test PASSED in 10.4s //tensorflow/python/tpu:tpu_test_wrapper_test PASSED in 10.7s //tensorflow/python/tpu/client:client_py_test PASSED in 10.5s //tensorflow/python/trackable:autotrackable_test PASSED in 10.4s //tensorflow/python/trackable:base_delegate_test PASSED in 11.0s //tensorflow/python/trackable:base_test PASSED in 10.1s //tensorflow/python/trackable:data_structures_test PASSED in 20.9s //tensorflow/python/trackable:python_state_test PASSED in 14.2s //tensorflow/python/trackable:resource_test PASSED in 11.5s //tensorflow/python/trackable:trackable_utils_test PASSED in 8.0s //tensorflow/python/training:adadelta_test_cpu PASSED in 20.9s //tensorflow/python/training:adagrad_da_test_cpu PASSED in 12.8s //tensorflow/python/training:adagrad_test_cpu PASSED in 16.9s //tensorflow/python/training:adam_test_cpu PASSED in 18.0s //tensorflow/python/training:basic_loops_test_cpu PASSED in 11.3s //tensorflow/python/training:basic_session_run_hooks_test PASSED in 36.0s //tensorflow/python/training:checkpoint_ops_test PASSED in 8.7s //tensorflow/python/training:coordinator_test_cpu PASSED in 18.3s //tensorflow/python/training:device_setter_test_cpu PASSED in 11.0s //tensorflow/python/training:ftrl_test_cpu PASSED in 22.6s //tensorflow/python/training:gradient_descent_test_cpu PASSED in 13.3s //tensorflow/python/training:input_test PASSED in 28.5s //tensorflow/python/training:momentum_test_cpu PASSED in 16.1s //tensorflow/python/training:monitored_session_test PASSED in 29.7s //tensorflow/python/training:moving_averages_test_cpu PASSED in 23.0s //tensorflow/python/training:optimizer_test_cpu PASSED in 14.4s //tensorflow/python/training:proximal_adagrad_test_cpu PASSED in 29.0s //tensorflow/python/training:proximal_gradient_descent_test_cpu PASSED in 12.7s //tensorflow/python/training:quantize_training_test_cpu PASSED in 10.6s //tensorflow/python/training:queue_runner_test_cpu PASSED in 11.4s //tensorflow/python/training:rmsprop_test_cpu PASSED in 28.9s //tensorflow/python/training:saver_large_partitioned_variable_test PASSED in 16.5s //tensorflow/python/training:saver_test_2gpu PASSED in 38.6s //tensorflow/python/training:saver_test_cpu PASSED in 41.8s //tensorflow/python/training:server_lib_multiple_containers_test PASSED in 16.5s //tensorflow/python/training:server_lib_same_variables_clear_container_test PASSED in 11.8s //tensorflow/python/training:server_lib_same_variables_clear_test PASSED in 10.2s //tensorflow/python/training:server_lib_same_variables_no_clear_test PASSED in 10.2s //tensorflow/python/training:server_lib_sparse_job_test PASSED in 13.3s //tensorflow/python/training:server_lib_test PASSED in 17.2s //tensorflow/python/training:session_manager_test_cpu PASSED in 87.9s //tensorflow/python/training:slot_creator_test_cpu PASSED in 15.2s //tensorflow/python/training:supervisor_test PASSED in 18.8s //tensorflow/python/training:training_ops_mlir_test_cpu PASSED in 11.3s //tensorflow/python/training:training_ops_test_cpu PASSED in 17.2s //tensorflow/python/training:training_util_test PASSED in 14.1s //tensorflow/python/training:warm_starting_util_test PASSED in 31.5s //tensorflow/python/training/experimental:loss_scale_optimizer_test PASSED in 22.7s //tensorflow/python/training/experimental:loss_scale_test PASSED in 30.1s //tensorflow/python/training/experimental:mixed_precision_test_cpu PASSED in 10.3s //tensorflow/python/training/saving:saveable_object_util_test PASSED in 10.7s //tensorflow/python/util:compat_test PASSED in 10.4s //tensorflow/python/util:decorator_utils_test PASSED in 9.4s //tensorflow/python/util:deprecation_test PASSED in 12.4s //tensorflow/python/util:dispatch_test PASSED in 11.1s //tensorflow/python/util:example_parser_configuration_test PASSED in 13.1s //tensorflow/python/util:fast_module_type_test PASSED in 11.5s //tensorflow/python/util:function_parameter_canonicalizer_test PASSED in 18.1s //tensorflow/python/util:function_utils_test PASSED in 9.5s //tensorflow/python/util:keyword_args_test PASSED in 18.6s //tensorflow/python/util:lazy_loader_test PASSED in 16.2s //tensorflow/python/util:lock_util_test PASSED in 12.9s //tensorflow/python/util:module_wrapper_test PASSED in 12.1s //tensorflow/python/util:nest_test PASSED in 32.7s //tensorflow/python/util:object_identity_test PASSED in 8.6s //tensorflow/python/util:pywrap_xla_ops_test PASSED in 5.2s //tensorflow/python/util:serialization_test PASSED in 9.1s //tensorflow/python/util:tf_contextlib_test PASSED in 9.7s //tensorflow/python/util:tf_decorator_test PASSED in 9.9s //tensorflow/python/util:tf_export_test PASSED in 13.8s //tensorflow/python/util:tf_inspect_test PASSED in 13.2s //tensorflow/python/util:tf_should_use_test PASSED in 15.9s //tensorflow/python/util:tf_stack_test PASSED in 11.0s //tensorflow/python/util:traceback_utils_test PASSED in 10.3s //tensorflow/python/util:type_annotations_test PASSED in 14.1s //tensorflow/python/util:variable_utils_test PASSED in 15.5s //tensorflow/python/util:vlog_test PASSED in 23.4s //tensorflow/tools/api/tests:module_test PASSED in 22.6s //tensorflow/tools/benchmark:benchmark_model_test PASSED in 2.1s //tensorflow/tools/common:public_api_test PASSED in 6.0s //tensorflow/tools/common:traverse_test PASSED in 5.9s //tensorflow/tools/compatibility:all_renames_v2_test PASSED in 10.4s //tensorflow/tools/compatibility:ast_edits_test PASSED in 10.0s //tensorflow/tools/compatibility:test_file_v1_0 PASSED in 21.9s //tensorflow/tools/compatibility:test_file_v2_0 PASSED in 39.3s //tensorflow/tools/compatibility:tf_upgrade_test PASSED in 8.7s //tensorflow/tools/compatibility:tf_upgrade_v2_safety_test PASSED in 10.9s //tensorflow/tools/docs:tf_doctest_test PASSED in 2.8s //tensorflow/tools/graph_transforms:file_utils_test PASSED in 0.6s //tensorflow/tools/graph_transforms:transform_graph_test PASSED in 1.7s //tensorflow/tools/graph_transforms:transform_utils_test PASSED in 1.8s //tensorflow/tools/graph_transforms:transforms_test PASSED in 2.8s //tensorflow/tools/proto_splitter:merge_test PASSED in 0.4s //tensorflow/tools/proto_splitter:split_graph_def_test PASSED in 8.6s //tensorflow/tools/proto_splitter:split_test PASSED in 10.1s //tensorflow/tools/proto_splitter:util_test PASSED in 12.0s //tensorflow/tools/proto_splitter/cc:composable_splitter_test PASSED in 0.2s //tensorflow/tools/proto_splitter/cc:graph_def_splitter_test PASSED in 0.6s //tensorflow/tools/proto_splitter/cc:saved_model_splitter_test PASSED in 0.2s //tensorflow/tools/proto_splitter/cc:util_test PASSED in 2.5s //tensorflow/tools/proto_splitter/python:saved_model_test PASSED in 10.4s //tensorflow/tools/proto_splitter/python:test_util_test PASSED in 9.0s //tensorflow/tools/proto_text:gen_proto_text_functions_lib_test PASSED in 0.1s //tensorflow/tools/tensorflow_builder/compat_checker:compat_checker_test PASSED in 0.6s //tensorflow/tsl/c:tsl_status_test PASSED in 0.1s //tensorflow/tsl/concurrency:async_value_ref_test PASSED in 0.6s //tensorflow/tsl/concurrency:async_value_test PASSED in 0.1s //tensorflow/tsl/concurrency:concurrent_vector_test PASSED in 0.1s //tensorflow/tsl/cuda:cudnn_version_test PASSED in 0.2s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_agent_test PASSED in 13.3s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_error_util_test PASSED in 0.2s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_recoverable_job_test PASSED in 1.7s //tensorflow/tsl/distributed_runtime/preemption:preemption_notifier_test PASSED in 4.7s //tensorflow/tsl/distributed_runtime/preemption:preemption_sync_manager_test PASSED in 6.0s //tensorflow/tsl/distributed_runtime/rpc:grpc_channel_test PASSED in 0.4s //tensorflow/tsl/distributed_runtime/rpc:grpc_util_test PASSED in 0.1s //tensorflow/tsl/framework:cancellation_test PASSED in 1.9s //tensorflow/tsl/framework:device_id_utils_test PASSED in 4.1s //tensorflow/tsl/framework/convolution:eigen_spatial_convolutions_test PASSED in 0.2s //tensorflow/tsl/lib/gtl:tsl_lib_gtl_tests PASSED in 0.1s //tensorflow/tsl/lib/hash:crc32c_test PASSED in 0.6s //tensorflow/tsl/lib/histogram:histogram_test PASSED in 0.8s //tensorflow/tsl/lib/io:buffered_file_test PASSED in 0.1s //tensorflow/tsl/lib/io:buffered_inputstream_test PASSED in 0.4s //tensorflow/tsl/lib/io:cache_test PASSED in 0.1s //tensorflow/tsl/lib/io:inputbuffer_test PASSED in 1.2s //tensorflow/tsl/lib/io:inputstream_interface_test PASSED in 0.1s //tensorflow/tsl/lib/io:random_inputstream_test PASSED in 0.3s //tensorflow/tsl/lib/io:record_reader_writer_test PASSED in 1.4s //tensorflow/tsl/lib/io:recordio_test PASSED in 0.2s //tensorflow/tsl/lib/io:table_test PASSED in 4.0s //tensorflow/tsl/lib/io:zlib_buffers_test PASSED in 3.5s //tensorflow/tsl/lib/io/snappy:snappy_test PASSED in 0.2s //tensorflow/tsl/lib/math:math_util_test PASSED in 0.1s //tensorflow/tsl/lib/random:distribution_sampler_test PASSED in 3.1s //tensorflow/tsl/lib/random:philox_random_test PASSED in 0.2s //tensorflow/tsl/lib/random:random_distributions_test PASSED in 20.4s //tensorflow/tsl/lib/random:simple_philox_test PASSED in 0.1s //tensorflow/tsl/lib/random:weighted_picker_test PASSED in 12.9s //tensorflow/tsl/platform:criticality_test PASSED in 0.1s //tensorflow/tsl/platform:ctstring_test PASSED in 0.1s //tensorflow/tsl/platform:denormal_test PASSED in 0.1s //tensorflow/tsl/platform:errors_test PASSED in 0.1s //tensorflow/tsl/platform:fingerprint_test PASSED in 0.1s //tensorflow/tsl/platform:hash_test PASSED in 0.8s //tensorflow/tsl/platform:integral_types_test PASSED in 0.1s //tensorflow/tsl/platform:intrusive_ptr_test PASSED in 0.3s //tensorflow/tsl/platform:logging_test PASSED in 41.0s //tensorflow/tsl/platform:mutex_test PASSED in 0.1s //tensorflow/tsl/platform:net_test PASSED in 0.2s //tensorflow/tsl/platform:numbers_test PASSED in 0.1s //tensorflow/tsl/platform:path_test PASSED in 0.1s //tensorflow/tsl/platform:port_test PASSED in 8.6s //tensorflow/tsl/platform:random_test PASSED in 2.2s //tensorflow/tsl/platform:refcount_test PASSED in 1.1s //tensorflow/tsl/platform:retrying_file_system_test PASSED in 0.2s //tensorflow/tsl/platform:retrying_utils_test PASSED in 0.1s //tensorflow/tsl/platform:scanner_test PASSED in 0.1s //tensorflow/tsl/platform:setround_test PASSED in 0.3s //tensorflow/tsl/platform:stacktrace_handler_test PASSED in 1.9s //tensorflow/tsl/platform:stacktrace_test PASSED in 0.1s //tensorflow/tsl/platform:status_matchers_test PASSED in 0.1s //tensorflow/tsl/platform:status_test PASSED in 0.1s //tensorflow/tsl/platform:statusor_test PASSED in 1.7s //tensorflow/tsl/platform:str_util_test PASSED in 0.1s //tensorflow/tsl/platform:strcat_test PASSED in 0.1s //tensorflow/tsl/platform:stringpiece_test PASSED in 0.5s //tensorflow/tsl/platform:stringprintf_test PASSED in 0.1s //tensorflow/tsl/platform:subprocess_test PASSED in 0.6s //tensorflow/tsl/platform:tstring_test PASSED in 0.1s //tensorflow/tsl/platform:unbounded_work_queue_test PASSED in 0.8s //tensorflow/tsl/platform/cloud:compute_engine_metadata_client_test PASSED in 0.1s //tensorflow/tsl/platform/cloud:compute_engine_zone_provider_test PASSED in 0.2s //tensorflow/tsl/platform/cloud:curl_http_request_test PASSED in 7.9s //tensorflow/tsl/platform/cloud:expiring_lru_cache_test PASSED in 0.1s //tensorflow/tsl/platform/cloud:gcs_dns_cache_test PASSED in 0.2s //tensorflow/tsl/platform/cloud:gcs_file_system_test PASSED in 5.2s //tensorflow/tsl/platform/cloud:gcs_throttle_test PASSED in 0.4s //tensorflow/tsl/platform/cloud:google_auth_provider_test PASSED in 0.2s //tensorflow/tsl/platform/cloud:oauth_client_test PASSED in 0.4s //tensorflow/tsl/platform/cloud:ram_file_block_cache_test PASSED in 2.2s //tensorflow/tsl/platform/cloud:time_util_test PASSED in 0.1s //tensorflow/tsl/profiler/backends/cpu:traceme_recorder_test PASSED in 0.1s //tensorflow/tsl/profiler/convert:trace_container_test PASSED in 0.1s //tensorflow/tsl/profiler/convert:trace_events_to_json_test PASSED in 0.1s //tensorflow/tsl/profiler/convert:xla_op_utils_test PASSED in 0.1s //tensorflow/tsl/profiler/convert:xplane_to_trace_events_test PASSED in 0.3s //tensorflow/tsl/profiler/lib:profiler_factory_test PASSED in 0.1s //tensorflow/tsl/profiler/lib:profiler_lock_test PASSED in 0.1s //tensorflow/tsl/profiler/lib:scoped_annotation_test PASSED in 0.1s //tensorflow/tsl/profiler/lib:traceme_encode_test PASSED in 0.1s //tensorflow/tsl/profiler/rpc/client:remote_profiler_session_manager_test PASSED in 3.5s //tensorflow/tsl/profiler/utils:buffer_pool_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:group_events_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:parse_annotation_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:preprocess_xplane_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:tf_op_utils_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:timespan_test PASSED in 0.2s //tensorflow/tsl/profiler/utils:tpu_xplane_utils_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:xplane_builder_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:xplane_utils_test PASSED in 0.1s //tensorflow/tsl/util:device_name_utils_test PASSED in 0.1s //tensorflow/tsl/util:stats_calculator_test PASSED in 0.1s //tensorflow/compiler/tests:complex_div_test_cpu PASSED in 9.8s Stats over 2 runs: max = 9.8s, min = 9.2s, avg = 9.5s, dev = 0.3s //tensorflow/compiler/tests:complex_div_test_cpu_mlir_bridge_test PASSED in 12.0s Stats over 2 runs: max = 12.0s, min = 11.1s, avg = 11.5s, dev = 0.4s //tensorflow/compiler/xla/tests:conditional_test_cpu PASSED in 9.9s Stats over 2 runs: max = 9.9s, min = 8.6s, avg = 9.3s, dev = 0.7s //tensorflow/python/data/experimental/kernel_tests/optimization:optimization_test PASSED in 22.2s Stats over 2 runs: max = 22.2s, min = 14.1s, avg = 18.2s, dev = 4.0s //tensorflow/python/data/experimental/kernel_tests/service:metadata_test PASSED in 21.1s Stats over 2 runs: max = 21.1s, min = 18.8s, avg = 19.9s, dev = 1.2s //tensorflow/python/data/kernel_tests:padded_batch_test PASSED in 38.0s Stats over 2 runs: max = 38.0s, min = 33.5s, avg = 35.7s, dev = 2.2s //tensorflow/python/data/kernel_tests:repeat_test PASSED in 32.9s Stats over 2 runs: max = 32.9s, min = 31.5s, avg = 32.2s, dev = 0.7s //tensorflow/python/data/kernel_tests:window_test PASSED in 80.2s Stats over 2 runs: max = 80.2s, min = 51.4s, avg = 65.8s, dev = 14.4s //tensorflow/python/kernel_tests/array_ops:scatter_nd_ops_test_cpu PASSED in 15.3s Stats over 2 runs: max = 15.3s, min = 14.9s, avg = 15.1s, dev = 0.2s //tensorflow/python/kernel_tests/control_flow:functional_ops_test_cpu PASSED in 19.9s Stats over 2 runs: max = 19.9s, min = 19.9s, avg = 19.9s, dev = 0.0s //tensorflow/python/kernel_tests/control_flow:map_fn_test_cpu PASSED in 12.6s Stats over 2 runs: max = 12.6s, min = 11.9s, avg = 12.2s, dev = 0.4s //tensorflow/python/kernel_tests/nn_ops:atrous_conv2d_test_cpu PASSED in 32.0s Stats over 2 runs: max = 32.0s, min = 19.9s, avg = 25.9s, dev = 6.1s //tensorflow/python/kernel_tests/nn_ops:bias_op_d9m_test_cpu PASSED in 116.0s Stats over 2 runs: max = 116.0s, min = 60.8s, avg = 88.4s, dev = 27.6s //tensorflow/python/kernel_tests/nn_ops:conv2d_backprop_filter_grad_test_cpu PASSED in 39.6s Stats over 2 runs: max = 39.6s, min = 10.6s, avg = 25.1s, dev = 14.5s //tensorflow/python/ops:control_flow_ops_test_cpu PASSED in 50.3s Stats over 2 runs: max = 50.3s, min = 45.5s, avg = 47.9s, dev = 2.4s //tensorflow/tsl/profiler/rpc/client:profiler_client_test FLAKY, failed in 1 out of 2 in 4.7s Stats over 2 runs: max = 4.7s, min = 3.7s, avg = 4.2s, dev = 0.5s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/tsl/profiler/rpc/client/profiler_client_test/test_attempts/attempt_1.log //tensorflow/compiler/tests:spacetobatch_op_test_cpu PASSED in 12.0s Stats over 3 runs: max = 12.0s, min = 11.2s, avg = 11.7s, dev = 0.4s //tensorflow/compiler/tests:spacetobatch_op_test_cpu_mlir_bridge_test PASSED in 16.7s Stats over 3 runs: max = 16.7s, min = 16.0s, avg = 16.3s, dev = 0.3s //tensorflow/compiler/xla/tests:triangular_solve_test_cpu PASSED in 68.1s Stats over 3 runs: max = 68.1s, min = 65.8s, avg = 66.7s, dev = 1.1s //tensorflow/core/data/service:thread_safe_buffer_test PASSED in 0.4s Stats over 3 runs: max = 0.4s, min = 0.3s, avg = 0.3s, dev = 0.1s //tensorflow/python/data/experimental/kernel_tests/service:multi_process_cluster_test PASSED in 21.9s Stats over 3 runs: max = 21.9s, min = 13.8s, avg = 18.3s, dev = 3.3s //tensorflow/python/data/kernel_tests:unique_test PASSED in 18.2s Stats over 3 runs: max = 18.2s, min = 15.9s, avg = 17.2s, dev = 0.9s //tensorflow/python/distribute/coordinator:metric_utils_test PASSED in 17.9s Stats over 3 runs: max = 17.9s, min = 14.4s, avg = 16.0s, dev = 1.5s //tensorflow/python/kernel_tests/array_ops:gather_op_test_cpu PASSED in 46.8s Stats over 3 runs: max = 46.8s, min = 45.1s, avg = 45.7s, dev = 0.8s //tensorflow/python/kernel_tests/array_ops:weights_broadcast_test PASSED in 11.5s Stats over 3 runs: max = 11.5s, min = 11.3s, avg = 11.3s, dev = 0.1s //tensorflow/python/kernel_tests/distributions:util_test_cpu PASSED in 23.2s Stats over 3 runs: max = 23.2s, min = 23.0s, avg = 23.1s, dev = 0.1s //tensorflow/python/kernel_tests/linalg:matrix_triangular_solve_op_test_cpu PASSED in 118.6s Stats over 3 runs: max = 118.6s, min = 12.5s, avg = 48.0s, dev = 49.9s //tensorflow/python/kernel_tests/random:multinomial_op_big_test_cpu PASSED in 20.6s Stats over 3 runs: max = 20.6s, min = 17.4s, avg = 18.4s, dev = 1.5s //tensorflow/compiler/xla/tests:dynamic_ops_test_cpu PASSED in 9.7s Stats over 4 runs: max = 9.7s, min = 9.0s, avg = 9.2s, dev = 0.3s //tensorflow/core/kernels:example_parsing_ops_test PASSED in 0.8s Stats over 4 runs: max = 0.8s, min = 0.6s, avg = 0.7s, dev = 0.1s //tensorflow/python/data/experimental/kernel_tests:auto_shard_dataset_test PASSED in 52.3s Stats over 4 runs: max = 52.3s, min = 19.3s, avg = 40.1s, dev = 12.4s //tensorflow/python/data/experimental/kernel_tests:map_and_batch_test PASSED in 79.2s Stats over 4 runs: max = 79.2s, min = 50.4s, avg = 63.1s, dev = 10.3s //tensorflow/python/data/experimental/kernel_tests:parse_example_dataset_test PASSED in 30.4s Stats over 4 runs: max = 30.4s, min = 14.3s, avg = 21.8s, dev = 7.1s //tensorflow/python/data/experimental/kernel_tests:rebatch_dataset_test PASSED in 24.9s Stats over 4 runs: max = 24.9s, min = 13.3s, avg = 17.6s, dev = 4.5s //tensorflow/python/data/experimental/kernel_tests:sql_dataset_test PASSED in 143.0s Stats over 4 runs: max = 143.0s, min = 131.4s, avg = 138.7s, dev = 4.4s //tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_ft_test PASSED in 11.6s Stats over 4 runs: max = 11.6s, min = 10.2s, avg = 10.9s, dev = 0.6s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_test PASSED in 56.3s Stats over 4 runs: max = 56.3s, min = 28.3s, avg = 42.7s, dev = 13.5s //tensorflow/python/data/kernel_tests:batch_test PASSED in 35.5s Stats over 4 runs: max = 35.5s, min = 27.1s, avg = 29.8s, dev = 3.3s //tensorflow/python/data/kernel_tests:fixed_length_record_dataset_test PASSED in 21.4s Stats over 4 runs: max = 21.4s, min = 14.3s, avg = 17.5s, dev = 3.1s //tensorflow/python/data/kernel_tests:from_generator_test PASSED in 30.4s Stats over 4 runs: max = 30.4s, min = 18.5s, avg = 24.0s, dev = 4.4s //tensorflow/python/data/kernel_tests:group_by_window_test PASSED in 70.3s Stats over 4 runs: max = 70.3s, min = 52.9s, avg = 60.4s, dev = 7.5s //tensorflow/python/data/kernel_tests:ragged_batch_test PASSED in 28.8s Stats over 4 runs: max = 28.8s, min = 26.0s, avg = 27.5s, dev = 1.3s //tensorflow/python/data/kernel_tests:skip_test PASSED in 36.6s Stats over 4 runs: max = 36.6s, min = 24.8s, avg = 31.0s, dev = 5.1s //tensorflow/python/data/kernel_tests:take_test PASSED in 56.5s Stats over 4 runs: max = 56.5s, min = 54.5s, avg = 55.6s, dev = 0.8s //tensorflow/python/data/kernel_tests:take_while_test PASSED in 29.9s Stats over 4 runs: max = 29.9s, min = 28.2s, avg = 29.3s, dev = 0.7s //tensorflow/python/data/kernel_tests:text_line_dataset_test PASSED in 34.4s Stats over 4 runs: max = 34.4s, min = 24.9s, avg = 29.1s, dev = 4.0s //tensorflow/python/data/kernel_tests:zip_test PASSED in 17.3s Stats over 4 runs: max = 17.3s, min = 13.4s, avg = 14.9s, dev = 1.6s //tensorflow/python/debug/lib:dumping_callback_test_cpu PASSED in 37.7s Stats over 4 runs: max = 37.7s, min = 36.9s, avg = 37.3s, dev = 0.3s //tensorflow/python/distribute:cross_device_ops_test_2gpu PASSED in 62.6s Stats over 4 runs: max = 62.6s, min = 36.1s, avg = 44.3s, dev = 10.7s //tensorflow/python/distribute:cross_device_ops_test_cpu PASSED in 35.1s Stats over 4 runs: max = 35.1s, min = 27.7s, avg = 30.4s, dev = 3.0s //tensorflow/python/framework:convert_to_constants_test PASSED in 29.3s Stats over 4 runs: max = 29.3s, min = 21.5s, avg = 24.7s, dev = 2.9s //tensorflow/python/kernel_tests:collective_ops_test_2gpu PASSED in 55.5s Stats over 4 runs: max = 55.5s, min = 53.0s, avg = 54.3s, dev = 0.9s //tensorflow/python/kernel_tests:collective_ops_test_cpu PASSED in 38.9s Stats over 4 runs: max = 38.9s, min = 36.4s, avg = 37.4s, dev = 0.9s //tensorflow/python/kernel_tests/array_ops:concat_op_test_cpu PASSED in 25.6s Stats over 4 runs: max = 25.6s, min = 24.9s, avg = 25.3s, dev = 0.3s //tensorflow/python/kernel_tests/array_ops:init_ops_test_cpu PASSED in 72.4s Stats over 4 runs: max = 72.4s, min = 29.3s, avg = 47.8s, dev = 17.6s //tensorflow/python/kernel_tests/array_ops:split_op_test_cpu PASSED in 36.8s Stats over 4 runs: max = 36.8s, min = 17.8s, avg = 25.2s, dev = 7.9s //tensorflow/python/kernel_tests/linalg:einsum_op_test_cpu PASSED in 78.3s Stats over 4 runs: max = 78.3s, min = 18.3s, avg = 43.9s, dev = 24.2s //tensorflow/python/kernel_tests/linalg:linear_operator_lower_triangular_test_cpu PASSED in 51.2s Stats over 4 runs: max = 51.2s, min = 48.7s, avg = 49.5s, dev = 1.0s //tensorflow/python/kernel_tests/nn_ops:conv_ops_test_cpu PASSED in 59.5s Stats over 4 runs: max = 59.5s, min = 33.1s, avg = 41.6s, dev = 10.5s //tensorflow/python/kernel_tests/random:random_gamma_test_cpu PASSED in 109.4s Stats over 4 runs: max = 109.4s, min = 11.2s, avg = 53.5s, dev = 42.8s //tensorflow/python/kernel_tests/signal:window_ops_test_cpu PASSED in 31.2s Stats over 4 runs: max = 31.2s, min = 30.0s, avg = 30.5s, dev = 0.4s //tensorflow/python/ops:nn_batchnorm_test_cpu PASSED in 21.6s Stats over 4 runs: max = 21.6s, min = 18.8s, avg = 20.2s, dev = 1.0s //tensorflow/python/ops:nn_fused_batchnorm_d9m_test_cpu PASSED in 36.8s Stats over 4 runs: max = 36.8s, min = 34.8s, avg = 35.8s, dev = 0.8s //tensorflow/python/ops/ragged:ragged_gather_op_test PASSED in 79.9s Stats over 4 runs: max = 79.9s, min = 20.8s, avg = 46.0s, dev = 21.4s //tensorflow/python/ops/ragged:ragged_getitem_test PASSED in 55.3s Stats over 4 runs: max = 55.3s, min = 50.2s, avg = 53.1s, dev = 1.9s //tensorflow/compiler/tests:async_comp_test_cpu PASSED in 14.4s Stats over 5 runs: max = 14.4s, min = 14.2s, avg = 14.3s, dev = 0.1s //tensorflow/compiler/tests:conv3d_test_cpu PASSED in 16.9s Stats over 5 runs: max = 16.9s, min = 9.2s, avg = 12.9s, dev = 3.2s //tensorflow/compiler/tests:conv3d_test_cpu_mlir_bridge_test PASSED in 17.2s Stats over 5 runs: max = 17.2s, min = 9.8s, avg = 13.2s, dev = 3.1s //tensorflow/compiler/tests:depthwise_conv_op_test_cpu PASSED in 16.8s Stats over 5 runs: max = 16.8s, min = 11.2s, avg = 13.8s, dev = 2.3s //tensorflow/compiler/tests:depthwise_conv_op_test_cpu_mlir_bridge_test PASSED in 46.2s Stats over 5 runs: max = 46.2s, min = 33.7s, avg = 39.0s, dev = 4.6s //tensorflow/compiler/tests:fused_batchnorm_test_cpu PASSED in 10.5s Stats over 5 runs: max = 10.5s, min = 9.9s, avg = 10.2s, dev = 0.2s //tensorflow/compiler/tests:fused_batchnorm_test_cpu_mlir_bridge_test PASSED in 27.3s Stats over 5 runs: max = 27.3s, min = 9.9s, avg = 23.7s, dev = 6.9s //tensorflow/compiler/tests:image_ops_jit_compile_test_cpu PASSED in 11.3s Stats over 5 runs: max = 11.3s, min = 9.5s, avg = 9.9s, dev = 0.7s //tensorflow/compiler/tests:reduce_ops_test_cpu PASSED in 14.1s Stats over 5 runs: max = 14.1s, min = 13.1s, avg = 13.6s, dev = 0.3s //tensorflow/compiler/tests:reduce_ops_test_cpu_mlir_bridge_test PASSED in 15.0s Stats over 5 runs: max = 15.0s, min = 13.7s, avg = 14.3s, dev = 0.5s //tensorflow/compiler/tests:repeat_op_test_cpu PASSED in 11.5s Stats over 5 runs: max = 11.5s, min = 8.9s, avg = 9.8s, dev = 1.0s //tensorflow/compiler/tests:repeat_op_test_cpu_mlir_bridge_test PASSED in 13.4s Stats over 5 runs: max = 13.4s, min = 11.4s, avg = 12.3s, dev = 0.7s //tensorflow/compiler/tests:special_math_test_cpu PASSED in 104.9s Stats over 5 runs: max = 104.9s, min = 17.1s, avg = 48.3s, dev = 30.2s //tensorflow/compiler/tests:special_math_test_cpu_mlir_bridge_test PASSED in 114.7s Stats over 5 runs: max = 114.7s, min = 28.9s, avg = 58.9s, dev = 30.0s //tensorflow/compiler/xla/client/lib:self_adjoint_eig_test_cpu PASSED in 25.7s Stats over 5 runs: max = 25.7s, min = 10.8s, avg = 19.3s, dev = 6.4s //tensorflow/core/grappler/optimizers:constant_folding_test PASSED in 2.5s Stats over 5 runs: max = 2.5s, min = 1.7s, avg = 2.2s, dev = 0.3s //tensorflow/dtensor/python/tests:layout_propagation_test_cpu PASSED in 23.1s Stats over 5 runs: max = 23.1s, min = 20.2s, avg = 22.3s, dev = 1.0s //tensorflow/dtensor/python/tests:multi_mesh_test_cpu PASSED in 13.1s Stats over 5 runs: max = 13.1s, min = 10.3s, avg = 11.8s, dev = 1.1s //tensorflow/python/distribute:mirrored_strategy_test_2gpu PASSED in 18.3s Stats over 5 runs: max = 18.3s, min = 14.4s, avg = 16.6s, dev = 1.3s //tensorflow/python/distribute:mirrored_strategy_test_cpu PASSED in 13.9s Stats over 5 runs: max = 13.9s, min = 12.9s, avg = 13.5s, dev = 0.3s //tensorflow/python/distribute:moving_averages_test_2gpu PASSED in 26.2s Stats over 5 runs: max = 26.2s, min = 21.5s, avg = 23.3s, dev = 1.6s //tensorflow/python/distribute:moving_averages_test_cpu PASSED in 27.4s Stats over 5 runs: max = 27.4s, min = 19.1s, avg = 22.7s, dev = 3.0s //tensorflow/python/distribute:vars_test_2gpu PASSED in 21.9s Stats over 5 runs: max = 21.9s, min = 19.2s, avg = 20.0s, dev = 1.0s //tensorflow/python/distribute:vars_test_cpu PASSED in 52.5s Stats over 5 runs: max = 52.5s, min = 27.3s, avg = 46.8s, dev = 9.8s //tensorflow/python/eager:device_placement_test_cpu PASSED in 10.6s Stats over 5 runs: max = 10.6s, min = 9.3s, avg = 10.0s, dev = 0.6s //tensorflow/python/eager:forwardprop_test_cpu PASSED in 117.7s Stats over 5 runs: max = 117.7s, min = 19.8s, avg = 58.6s, dev = 32.5s //tensorflow/python/eager/polymorphic_function:gradients_test_cpu PASSED in 29.3s Stats over 5 runs: max = 29.3s, min = 22.7s, avg = 25.9s, dev = 2.7s //tensorflow/python/kernel_tests/linalg:cholesky_op_test_cpu PASSED in 53.6s Stats over 5 runs: max = 53.6s, min = 34.9s, avg = 44.3s, dev = 6.2s //tensorflow/python/kernel_tests/linalg:linear_operator_adjoint_test_cpu PASSED in 45.3s Stats over 5 runs: max = 45.3s, min = 44.2s, avg = 44.8s, dev = 0.4s //tensorflow/python/kernel_tests/linalg:linear_operator_composition_test_cpu PASSED in 50.6s Stats over 5 runs: max = 50.6s, min = 48.7s, avg = 50.1s, dev = 0.7s //tensorflow/python/kernel_tests/linalg:linear_operator_diag_test_cpu PASSED in 43.0s Stats over 5 runs: max = 43.0s, min = 39.3s, avg = 40.3s, dev = 1.4s //tensorflow/python/kernel_tests/linalg:linear_operator_full_matrix_test_cpu PASSED in 53.1s Stats over 5 runs: max = 53.1s, min = 51.6s, avg = 52.4s, dev = 0.4s //tensorflow/python/kernel_tests/linalg:linear_operator_householder_test_cpu PASSED in 53.6s Stats over 5 runs: max = 53.6s, min = 41.4s, avg = 48.7s, dev = 4.5s //tensorflow/python/kernel_tests/linalg:linear_operator_identity_test_cpu PASSED in 35.0s Stats over 5 runs: max = 35.0s, min = 34.5s, avg = 34.8s, dev = 0.2s //tensorflow/python/kernel_tests/linalg:linear_operator_inversion_test_cpu PASSED in 30.9s Stats over 5 runs: max = 30.9s, min = 29.6s, avg = 30.3s, dev = 0.5s //tensorflow/python/kernel_tests/linalg:linear_operator_permutation_test_cpu PASSED in 29.8s Stats over 5 runs: max = 29.8s, min = 27.8s, avg = 28.4s, dev = 0.7s //tensorflow/python/kernel_tests/linalg:linear_operator_toeplitz_test_cpu PASSED in 31.0s Stats over 5 runs: max = 31.0s, min = 29.4s, avg = 30.1s, dev = 0.5s //tensorflow/python/kernel_tests/linalg:linear_operator_tridiag_test_cpu PASSED in 128.1s Stats over 5 runs: max = 128.1s, min = 120.5s, avg = 123.6s, dev = 2.7s //tensorflow/python/kernel_tests/linalg:linear_operator_util_test_cpu PASSED in 14.4s Stats over 5 runs: max = 14.4s, min = 13.5s, avg = 14.1s, dev = 0.3s //tensorflow/python/kernel_tests/linalg:linear_operator_zeros_test_cpu PASSED in 20.0s Stats over 5 runs: max = 20.0s, min = 18.9s, avg = 19.4s, dev = 0.4s //tensorflow/python/kernel_tests/nn_ops:fractional_avg_pool_op_test PASSED in 16.3s Stats over 5 runs: max = 16.3s, min = 9.1s, avg = 11.0s, dev = 2.7s //tensorflow/python/kernel_tests/nn_ops:fractional_max_pool_op_test PASSED in 8.7s Stats over 5 runs: max = 8.7s, min = 3.2s, avg = 5.5s, dev = 2.7s //tensorflow/python/kernel_tests/sparse_ops:sparse_ops_test_cpu PASSED in 30.5s Stats over 5 runs: max = 30.5s, min = 8.1s, avg = 14.0s, dev = 8.4s //tensorflow/python/ops/parallel_for:math_test_cpu PASSED in 86.2s Stats over 5 runs: max = 86.2s, min = 29.2s, avg = 53.5s, dev = 21.0s //tensorflow/compiler/tests:scan_ops_test_cpu PASSED in 17.3s Stats over 6 runs: max = 17.3s, min = 13.5s, avg = 15.3s, dev = 1.2s //tensorflow/compiler/tests:scan_ops_test_cpu_mlir_bridge_test PASSED in 20.1s Stats over 6 runs: max = 20.1s, min = 14.2s, avg = 17.0s, dev = 1.7s //tensorflow/python/data/experimental/kernel_tests:make_batched_features_dataset_test PASSED in 28.8s Stats over 6 runs: max = 28.8s, min = 8.3s, avg = 16.8s, dev = 7.9s //tensorflow/python/kernel_tests/array_ops:diag_op_test_cpu PASSED in 55.7s Stats over 6 runs: max = 55.7s, min = 12.0s, avg = 21.7s, dev = 15.3s //tensorflow/python/kernel_tests/math_ops:reduction_ops_test_cpu PASSED in 64.1s Stats over 6 runs: max = 64.1s, min = 26.1s, avg = 47.6s, dev = 11.6s //tensorflow/python/ops:accumulate_n_benchmark_cpu PASSED in 13.4s Stats over 6 runs: max = 13.4s, min = 11.9s, avg = 12.8s, dev = 0.6s //tensorflow/python/distribute/experimental/rpc:rpc_ops_test PASSED in 14.9s Stats over 7 runs: max = 14.9s, min = 10.6s, avg = 12.2s, dev = 1.5s //tensorflow/compiler/tests:matrix_diag_ops_test_cpu PASSED in 60.8s Stats over 8 runs: max = 60.8s, min = 8.3s, avg = 25.4s, dev = 17.4s //tensorflow/compiler/tests:matrix_diag_ops_test_cpu_mlir_bridge_test PASSED in 120.8s Stats over 8 runs: max = 120.8s, min = 7.7s, avg = 47.6s, dev = 39.6s //tensorflow/dtensor/python/tests:input_util_test PASSED in 27.6s Stats over 8 runs: max = 27.6s, min = 17.9s, avg = 22.9s, dev = 3.0s //tensorflow/python/data/experimental/kernel_tests:csv_dataset_test PASSED in 34.2s Stats over 8 runs: max = 34.2s, min = 10.5s, avg = 19.3s, dev = 9.2s //tensorflow/python/data/experimental/kernel_tests:parallel_interleave_test PASSED in 32.0s Stats over 8 runs: max = 32.0s, min = 13.9s, avg = 22.1s, dev = 6.2s //tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_ft_test PASSED in 47.1s Stats over 8 runs: max = 47.1s, min = 8.2s, avg = 25.8s, dev = 16.8s //tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_test PASSED in 48.4s Stats over 8 runs: max = 48.4s, min = 9.4s, avg = 20.6s, dev = 13.8s //tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_test PASSED in 31.3s Stats over 8 runs: max = 31.3s, min = 10.2s, avg = 18.1s, dev = 8.1s //tensorflow/python/data/experimental/kernel_tests/service:fault_tolerance_test PASSED in 19.9s Stats over 8 runs: max = 19.9s, min = 10.5s, avg = 13.5s, dev = 3.5s //tensorflow/python/data/kernel_tests:filter_test PASSED in 23.1s Stats over 8 runs: max = 23.1s, min = 17.5s, avg = 19.8s, dev = 1.7s //tensorflow/python/data/kernel_tests:flat_map_test PASSED in 27.9s Stats over 8 runs: max = 27.9s, min = 16.5s, avg = 20.9s, dev = 4.2s //tensorflow/python/data/kernel_tests:shard_test PASSED in 42.0s Stats over 8 runs: max = 42.0s, min = 33.9s, avg = 37.9s, dev = 2.7s //tensorflow/python/data/kernel_tests:shuffle_test PASSED in 65.9s Stats over 8 runs: max = 65.9s, min = 31.8s, avg = 37.2s, dev = 10.9s //tensorflow/python/data/kernel_tests:tf_record_dataset_test PASSED in 32.6s Stats over 8 runs: max = 32.6s, min = 15.9s, avg = 24.9s, dev = 4.4s //tensorflow/python/distribute/failure_handling:gce_failure_handler_test PASSED in 141.9s Stats over 8 runs: max = 141.9s, min = 16.5s, avg = 57.1s, dev = 49.8s //tensorflow/python/kernel_tests/linalg:linalg_ops_test_cpu PASSED in 64.1s Stats over 8 runs: max = 64.1s, min = 39.8s, avg = 53.8s, dev = 8.5s //tensorflow/python/kernel_tests/linalg:linear_operator_block_diag_test_cpu PASSED in 114.3s Stats over 8 runs: max = 114.3s, min = 94.4s, avg = 107.2s, dev = 6.9s //tensorflow/python/kernel_tests/linalg:linear_operator_block_lower_triangular_test_cpu PASSED in 79.2s Stats over 8 runs: max = 79.2s, min = 56.3s, avg = 68.7s, dev = 7.8s //tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_d9m_test_cpu PASSED in 87.0s Stats over 8 runs: max = 87.0s, min = 8.3s, avg = 21.0s, dev = 25.4s //tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_test_cpu PASSED in 10.3s Stats over 8 runs: max = 10.3s, min = 9.2s, avg = 9.7s, dev = 0.4s //tensorflow/python/kernel_tests/signal:fft_ops_test_cpu PASSED in 18.3s Stats over 8 runs: max = 18.3s, min = 10.7s, avg = 14.4s, dev = 3.4s //tensorflow/python/ops/ragged:dynamic_ragged_shape_test PASSED in 67.5s Stats over 8 runs: max = 67.5s, min = 52.6s, avg = 57.6s, dev = 5.4s //tensorflow/python/ops/ragged:ragged_tensor_test PASSED in 26.0s Stats over 8 runs: max = 26.0s, min = 15.2s, avg = 19.5s, dev = 3.0s //tensorflow/python/distribute/failure_handling:failure_handler_test FLAKY, failed in 1 out of 9 in 64.5s Stats over 9 runs: max = 64.5s, min = 25.9s, avg = 46.7s, dev = 12.3s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/distribute/failure_handling/failure_handler_test/shard_1_of_8/test_attempts/attempt_1.log //tensorflow/compiler/tests:bincount_op_test_cpu PASSED in 8.8s Stats over 10 runs: max = 8.8s, min = 4.5s, avg = 6.7s, dev = 1.8s //tensorflow/compiler/tests:conv2d_test_cpu PASSED in 9.3s Stats over 10 runs: max = 9.3s, min = 5.0s, avg = 6.6s, dev = 1.5s //tensorflow/compiler/tests:conv2d_test_cpu_mlir_bridge_test PASSED in 8.4s Stats over 10 runs: max = 8.4s, min = 3.2s, avg = 4.6s, dev = 1.9s //tensorflow/compiler/tests:random_ops_test_cpu PASSED in 15.0s Stats over 10 runs: max = 15.0s, min = 7.6s, avg = 10.9s, dev = 2.1s //tensorflow/compiler/tests:random_ops_test_cpu_mlir_bridge_test PASSED in 16.2s Stats over 10 runs: max = 16.2s, min = 9.2s, avg = 12.8s, dev = 2.3s //tensorflow/compiler/tests:stateless_random_ops_test_cpu PASSED in 83.1s Stats over 10 runs: max = 83.1s, min = 45.7s, avg = 64.6s, dev = 11.0s //tensorflow/compiler/tests:stateless_random_ops_test_cpu_mlir_bridge_test PASSED in 84.5s Stats over 10 runs: max = 84.5s, min = 47.6s, avg = 63.3s, dev = 11.2s //tensorflow/compiler/tests:stochastic_cast_op_test_cpu PASSED in 23.9s Stats over 10 runs: max = 23.9s, min = 15.0s, avg = 20.2s, dev = 2.9s //tensorflow/compiler/xla/client/lib:svd_test_cpu PASSED in 26.2s Stats over 10 runs: max = 26.2s, min = 6.4s, avg = 14.9s, dev = 8.6s //tensorflow/compiler/xla/client/lib:tridiagonal_test_cpu PASSED in 11.2s Stats over 10 runs: max = 11.2s, min = 8.6s, avg = 9.8s, dev = 1.0s //tensorflow/compiler/xla/service/cpu:cpu_runtime_test PASSED in 14.3s Stats over 10 runs: max = 14.3s, min = 0.9s, avg = 10.6s, dev = 4.9s //tensorflow/python/data/kernel_tests:rejection_resample_test PASSED in 15.9s Stats over 10 runs: max = 15.9s, min = 7.5s, avg = 11.6s, dev = 2.3s //tensorflow/python/distribute:input_lib_type_spec_test_2gpu PASSED in 26.7s Stats over 10 runs: max = 26.7s, min = 14.9s, avg = 20.6s, dev = 3.8s //tensorflow/python/distribute:input_lib_type_spec_test_cpu PASSED in 19.7s Stats over 10 runs: max = 19.7s, min = 12.1s, avg = 15.8s, dev = 2.5s //tensorflow/python/framework:config_vgpu_test_2gpu PASSED in 9.7s Stats over 10 runs: max = 9.7s, min = 8.8s, avg = 9.2s, dev = 0.2s //tensorflow/python/framework:config_vgpu_test_cpu PASSED in 10.9s Stats over 10 runs: max = 10.9s, min = 9.2s, avg = 9.8s, dev = 0.5s //tensorflow/python/framework:function_test_cpu PASSED in 56.0s Stats over 10 runs: max = 56.0s, min = 11.0s, avg = 18.7s, dev = 12.7s //tensorflow/python/grappler:cluster_test_cpu PASSED in 10.5s Stats over 10 runs: max = 10.5s, min = 8.5s, avg = 9.5s, dev = 0.7s //tensorflow/python/kernel_tests/array_ops:array_ops_test_cpu PASSED in 23.0s Stats over 10 runs: max = 23.0s, min = 15.8s, avg = 19.9s, dev = 2.3s //tensorflow/python/kernel_tests/array_ops:inplace_ops_test_cpu PASSED in 10.2s Stats over 10 runs: max = 10.2s, min = 8.6s, avg = 9.5s, dev = 0.5s //tensorflow/python/kernel_tests/data_structures:tensor_array_ops_test_cpu PASSED in 9.9s Stats over 10 runs: max = 9.9s, min = 4.8s, avg = 6.5s, dev = 1.8s //tensorflow/python/kernel_tests/linalg:linear_operator_low_rank_update_test_cpu PASSED in 115.4s Stats over 10 runs: max = 115.4s, min = 102.4s, avg = 109.1s, dev = 3.4s //tensorflow/python/kernel_tests/linalg:tridiagonal_matmul_op_test_cpu PASSED in 145.1s Stats over 10 runs: max = 145.1s, min = 7.9s, avg = 22.9s, dev = 40.8s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_ops_test_cpu PASSED in 34.2s Stats over 10 runs: max = 34.2s, min = 9.0s, avg = 19.3s, dev = 7.4s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_test_cpu PASSED in 25.1s Stats over 10 runs: max = 25.1s, min = 9.1s, avg = 15.5s, dev = 6.3s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_test_cpu PASSED in 34.0s Stats over 10 runs: max = 34.0s, min = 14.8s, avg = 19.8s, dev = 7.2s //tensorflow/python/kernel_tests/nn_ops:rnn_test_cpu PASSED in 17.5s Stats over 10 runs: max = 17.5s, min = 13.7s, avg = 16.0s, dev = 1.0s //tensorflow/python/kernel_tests/random:random_index_shuffle_test PASSED in 26.4s Stats over 10 runs: max = 26.4s, min = 23.6s, avg = 24.9s, dev = 0.8s //tensorflow/python/kernel_tests/random:stateless_random_ops_test_cpu PASSED in 130.6s Stats over 10 runs: max = 130.6s, min = 22.6s, avg = 73.0s, dev = 49.3s //tensorflow/python/ops:special_math_ops_test_cpu PASSED in 47.3s Stats over 10 runs: max = 47.3s, min = 9.5s, avg = 16.3s, dev = 10.6s //tensorflow/python/ops:weak_tensor_special_math_ops_test_cpu PASSED in 17.3s Stats over 10 runs: max = 17.3s, min = 11.6s, avg = 14.3s, dev = 1.6s //tensorflow/python/ops/numpy_ops/tests:np_indexing_test PASSED in 132.0s Stats over 10 runs: max = 132.0s, min = 119.9s, avg = 128.3s, dev = 3.3s //tensorflow/python/ops/ragged:ragged_tensor_supported_values_test PASSED in 40.5s Stats over 10 runs: max = 40.5s, min = 19.5s, avg = 22.9s, dev = 5.9s //tensorflow/python/saved_model:load_test_cpu PASSED in 61.6s Stats over 10 runs: max = 61.6s, min = 35.3s, avg = 43.1s, dev = 6.9s //tensorflow/compiler/tests:fft_test_cpu PASSED in 29.9s Stats over 12 runs: max = 29.9s, min = 13.4s, avg = 19.3s, dev = 5.7s //tensorflow/compiler/xla/service:triangular_solve_expander_test PASSED in 4.4s Stats over 12 runs: max = 4.4s, min = 1.9s, avg = 3.0s, dev = 0.7s //tensorflow/python/data/experimental/kernel_tests:group_by_reducer_test PASSED in 20.3s Stats over 12 runs: max = 20.3s, min = 4.9s, avg = 12.9s, dev = 4.0s //tensorflow/python/data/kernel_tests:choose_from_datasets_test PASSED in 24.1s Stats over 12 runs: max = 24.1s, min = 18.5s, avg = 20.3s, dev = 1.7s //tensorflow/python/data/kernel_tests:memory_cleanup_test_cpu PASSED in 14.8s Stats over 12 runs: max = 14.8s, min = 8.9s, avg = 11.8s, dev = 1.8s //tensorflow/python/distribute:multi_process_runner_test_2gpu PASSED in 255.7s Stats over 12 runs: max = 255.7s, min = 26.8s, avg = 78.7s, dev = 58.3s //tensorflow/python/distribute:multi_process_runner_test_cpu PASSED in 233.5s Stats over 12 runs: max = 233.5s, min = 16.3s, avg = 54.7s, dev = 59.8s //tensorflow/python/eager/polymorphic_function:polymorphic_function_test_cpu PASSED in 21.1s Stats over 15 runs: max = 21.1s, min = 10.1s, avg = 14.3s, dev = 3.0s //tensorflow/python/kernel_tests/nn_ops:rnn_cell_test_cpu PASSED in 58.6s Stats over 15 runs: max = 58.6s, min = 15.3s, avg = 20.7s, dev = 11.1s //tensorflow/compiler/tests:ftrl_test_cpu PASSED in 11.0s Stats over 16 runs: max = 11.0s, min = 6.2s, avg = 8.3s, dev = 1.4s //tensorflow/compiler/tests:ternary_ops_test_cpu PASSED in 16.8s Stats over 16 runs: max = 16.8s, min = 5.1s, avg = 10.8s, dev = 2.9s //tensorflow/compiler/tests:ternary_ops_test_cpu_mlir_bridge_test PASSED in 24.8s Stats over 16 runs: max = 24.8s, min = 12.9s, avg = 15.3s, dev = 2.9s //tensorflow/python/data/experimental/kernel_tests/service:dynamic_sharding_test PASSED in 20.4s Stats over 16 runs: max = 20.4s, min = 10.5s, avg = 15.0s, dev = 3.5s //tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test PASSED in 31.2s Stats over 16 runs: max = 31.2s, min = 5.5s, avg = 13.9s, dev = 6.1s //tensorflow/python/data/kernel_tests:snapshot_test PASSED in 30.5s Stats over 16 runs: max = 30.5s, min = 12.4s, avg = 20.6s, dev = 4.7s //tensorflow/python/kernel_tests/control_flow:control_flow_ops_py_test_cpu PASSED in 25.3s Stats over 16 runs: max = 25.3s, min = 5.3s, avg = 9.6s, dev = 4.8s //tensorflow/python/kernel_tests/linalg:matrix_exponential_op_test PASSED in 13.6s Stats over 16 runs: max = 13.6s, min = 6.7s, avg = 9.6s, dev = 1.7s //tensorflow/python/kernel_tests/signal:dct_ops_test_cpu PASSED in 15.6s Stats over 16 runs: max = 15.6s, min = 11.1s, avg = 13.3s, dev = 1.2s //tensorflow/python/ops:image_ops_test_cpu PASSED in 27.9s Stats over 16 runs: max = 27.9s, min = 7.4s, avg = 14.4s, dev = 6.5s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test PASSED in 92.1s Stats over 17 runs: max = 92.1s, min = 5.7s, avg = 27.8s, dev = 34.8s //tensorflow/python/data/kernel_tests:map_test PASSED in 38.2s Stats over 19 runs: max = 38.2s, min = 10.4s, avg = 21.6s, dev = 6.1s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu PASSED in 10.2s Stats over 20 runs: max = 10.2s, min = 5.9s, avg = 8.2s, dev = 1.2s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu_mlir_bridge_test PASSED in 13.6s Stats over 20 runs: max = 13.6s, min = 11.3s, avg = 12.6s, dev = 0.7s //tensorflow/compiler/tests:pooling_ops_test_cpu PASSED in 13.4s Stats over 20 runs: max = 13.4s, min = 5.1s, avg = 7.8s, dev = 2.0s //tensorflow/compiler/tests:pooling_ops_test_cpu_mlir_bridge_test PASSED in 11.6s Stats over 20 runs: max = 11.6s, min = 4.8s, avg = 7.9s, dev = 2.0s //tensorflow/compiler/xla/tests:convolution_dimension_numbers_test_cpu PASSED in 9.8s Stats over 20 runs: max = 9.8s, min = 7.8s, avg = 8.8s, dev = 0.5s //tensorflow/compiler/xla/tests:dot_operation_single_threaded_runtime_test_cpu PASSED in 15.3s Stats over 20 runs: max = 15.3s, min = 8.7s, avg = 12.1s, dev = 1.6s //tensorflow/compiler/xla/tests:dot_operation_test_cpu PASSED in 12.4s Stats over 20 runs: max = 12.4s, min = 10.2s, avg = 10.9s, dev = 0.5s //tensorflow/compiler/xla/tests:prng_test_cpu PASSED in 8.0s Stats over 20 runs: max = 8.0s, min = 6.0s, avg = 6.9s, dev = 0.5s //tensorflow/compiler/xla/tests:reduce_window_test_cpu PASSED in 54.7s Stats over 20 runs: max = 54.7s, min = 6.5s, avg = 16.2s, dev = 12.2s //tensorflow/python/autograph/tests:loop_control_flow_test PASSED in 36.6s Stats over 20 runs: max = 36.6s, min = 32.7s, avg = 34.6s, dev = 1.1s //tensorflow/python/kernel_tests:metrics_test PASSED in 49.5s Stats over 20 runs: max = 49.5s, min = 12.9s, avg = 23.2s, dev = 10.5s //tensorflow/python/kernel_tests/array_ops:matrix_band_part_op_test_cpu PASSED in 18.9s Stats over 20 runs: max = 18.9s, min = 16.5s, avg = 18.1s, dev = 0.8s //tensorflow/python/kernel_tests/data_structures:barrier_ops_test PASSED in 13.6s Stats over 20 runs: max = 13.6s, min = 5.2s, avg = 8.9s, dev = 2.1s //tensorflow/python/kernel_tests/linalg:eig_op_test PASSED in 48.9s Stats over 20 runs: max = 48.9s, min = 8.5s, avg = 20.1s, dev = 12.6s //tensorflow/python/kernel_tests/linalg:linalg_grad_test_cpu PASSED in 113.9s Stats over 20 runs: max = 113.9s, min = 46.7s, avg = 68.5s, dev = 20.9s //tensorflow/python/kernel_tests/linalg:norm_op_test_cpu PASSED in 12.0s Stats over 20 runs: max = 12.0s, min = 5.1s, avg = 8.6s, dev = 2.0s //tensorflow/python/kernel_tests/linalg:normalize_op_test_cpu PASSED in 15.4s Stats over 20 runs: max = 15.4s, min = 6.1s, avg = 11.4s, dev = 2.3s //tensorflow/python/kernel_tests/linalg:qr_op_test_cpu PASSED in 214.0s Stats over 20 runs: max = 214.0s, min = 68.3s, avg = 127.3s, dev = 46.0s //tensorflow/python/kernel_tests/linalg:self_adjoint_eig_op_test_cpu PASSED in 28.8s Stats over 20 runs: max = 28.8s, min = 11.3s, avg = 17.1s, dev = 5.3s //tensorflow/python/kernel_tests/math_ops:batch_matmul_op_test_cpu PASSED in 25.4s Stats over 20 runs: max = 25.4s, min = 8.1s, avg = 16.5s, dev = 6.2s //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu PASSED in 52.0s Stats over 20 runs: max = 52.0s, min = 31.8s, avg = 40.2s, dev = 7.9s //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu PASSED in 70.9s Stats over 20 runs: max = 70.9s, min = 10.9s, avg = 33.1s, dev = 20.4s //tensorflow/python/kernel_tests/nn_ops:embedding_ops_test_cpu PASSED in 24.2s Stats over 20 runs: max = 24.2s, min = 12.4s, avg = 14.8s, dev = 2.5s //tensorflow/python/data/kernel_tests:interleave_test PASSED in 32.2s Stats over 24 runs: max = 32.2s, min = 11.8s, avg = 20.6s, dev = 5.9s //tensorflow/python/data/kernel_tests:sample_from_datasets_test PASSED in 27.2s Stats over 24 runs: max = 27.2s, min = 8.8s, avg = 16.4s, dev = 4.8s //tensorflow/compiler/xla/tests:array_elementwise_ops_test_cpu PASSED in 9.3s Stats over 25 runs: max = 9.3s, min = 7.2s, avg = 8.3s, dev = 0.6s //tensorflow/compiler/xla/tests:select_and_scatter_test_cpu PASSED in 36.3s Stats over 25 runs: max = 36.3s, min = 7.2s, avg = 11.7s, dev = 7.5s //tensorflow/python/data/experimental/kernel_tests/service:local_workers_test FLAKY, failed in 1 out of 25 in 900.1s Stats over 25 runs: max = 900.1s, min = 14.2s, avg = 59.2s, dev = 171.7s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/data/experimental/kernel_tests/service/local_workers_test/shard_1_of_24/test_attempts/attempt_1.log //tensorflow/compiler/xla/tests:convolution_variants_test_cpu PASSED in 9.6s Stats over 30 runs: max = 9.6s, min = 8.0s, avg = 9.0s, dev = 0.4s //tensorflow/compiler/xla/tests:iota_test_cpu PASSED in 29.5s Stats over 30 runs: max = 29.5s, min = 11.3s, avg = 15.2s, dev = 6.8s //tensorflow/compiler/xla/tests:params_test_cpu PASSED in 9.3s Stats over 30 runs: max = 9.3s, min = 7.1s, avg = 8.2s, dev = 0.6s //tensorflow/compiler/xla/tests:reshape_test_cpu PASSED in 8.3s Stats over 30 runs: max = 8.3s, min = 6.0s, avg = 7.3s, dev = 0.7s //tensorflow/python/kernel_tests/nn_ops:conv_ops_3d_test_cpu PASSED in 18.6s Stats over 30 runs: max = 18.6s, min = 6.7s, avg = 11.5s, dev = 3.1s //tensorflow/compiler/xla/tests:reduce_test_cpu PASSED in 8.7s Stats over 31 runs: max = 8.7s, min = 6.9s, avg = 7.7s, dev = 0.5s //tensorflow/compiler/xla/tests:scalar_computations_test_cpu PASSED in 10.8s Stats over 32 runs: max = 10.8s, min = 6.8s, avg = 8.2s, dev = 0.9s //tensorflow/python/data/experimental/kernel_tests/service:data_service_ops_test PASSED in 20.5s Stats over 32 runs: max = 20.5s, min = 4.9s, avg = 9.7s, dev = 4.1s //tensorflow/python/kernel_tests/linalg:linear_operator_circulant_test_cpu PASSED in 43.2s Stats over 32 runs: max = 43.2s, min = 32.6s, avg = 37.8s, dev = 2.8s //tensorflow/compiler/xla/tests:batch_normalization_test_cpu PASSED in 10.4s Stats over 40 runs: max = 10.4s, min = 7.3s, avg = 9.1s, dev = 0.9s //tensorflow/compiler/xla/tests:bfloat16_test_cpu PASSED in 10.0s Stats over 40 runs: max = 10.0s, min = 6.6s, avg = 7.4s, dev = 0.7s //tensorflow/compiler/xla/tests:conv_depthwise_backprop_filter_test_cpu PASSED in 10.2s Stats over 40 runs: max = 10.2s, min = 7.4s, avg = 8.6s, dev = 0.7s //tensorflow/compiler/xla/tests:slice_test_cpu PASSED in 20.1s Stats over 40 runs: max = 20.1s, min = 8.0s, avg = 11.8s, dev = 4.0s //tensorflow/core/kernels:stochastic_cast_op_test PASSED in 1.7s Stats over 48 runs: max = 1.7s, min = 0.5s, avg = 0.7s, dev = 0.3s //tensorflow/compiler/mlir/quantization/tensorflow/python:quantize_model_test PASSED in 54.8s Stats over 50 runs: max = 54.8s, min = 19.8s, avg = 40.8s, dev = 7.3s //tensorflow/compiler/tests:sort_ops_test_cpu PASSED in 49.1s Stats over 50 runs: max = 49.1s, min = 3.7s, avg = 15.9s, dev = 9.7s //tensorflow/compiler/tests:sort_ops_test_cpu_mlir_bridge_test PASSED in 41.9s Stats over 50 runs: max = 41.9s, min = 3.6s, avg = 11.3s, dev = 8.3s //tensorflow/compiler/tests:unary_ops_test_cpu PASSED in 38.8s Stats over 50 runs: max = 38.8s, min = 4.5s, avg = 9.2s, dev = 6.9s //tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test PASSED in 25.9s Stats over 50 runs: max = 25.9s, min = 3.5s, avg = 7.2s, dev = 5.0s //tensorflow/compiler/xla/tests:conv_depthwise_test_cpu PASSED in 13.0s Stats over 50 runs: max = 13.0s, min = 6.9s, avg = 10.5s, dev = 1.9s //tensorflow/compiler/xla/tests:convolution_test_1d_no_vmodule_cpu PASSED in 14.0s Stats over 50 runs: max = 14.0s, min = 11.2s, avg = 12.3s, dev = 0.8s //tensorflow/compiler/xla/tests:convolution_test_cpu PASSED in 13.6s Stats over 50 runs: max = 13.6s, min = 8.3s, avg = 10.5s, dev = 1.2s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_grad_test_cpu PASSED in 15.9s Stats over 50 runs: max = 15.9s, min = 5.7s, avg = 10.1s, dev = 2.7s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_grad_test_cpu PASSED in 13.0s Stats over 50 runs: max = 13.0s, min = 3.8s, avg = 9.2s, dev = 2.2s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_sparse_mat_mul_grad_test_cpu PASSED in 33.4s Stats over 50 runs: max = 33.4s, min = 3.8s, avg = 15.2s, dev = 12.3s //tensorflow/python/kernel_tests/math_ops:cwise_ops_binary_test_cpu PASSED in 30.8s Stats over 50 runs: max = 30.8s, min = 7.5s, avg = 16.9s, dev = 6.4s //tensorflow/python/kernel_tests/math_ops:cwise_ops_test_cpu PASSED in 10.5s Stats over 50 runs: max = 10.5s, min = 4.2s, avg = 7.3s, dev = 1.9s //tensorflow/python/kernel_tests/math_ops:cwise_ops_unary_test_cpu PASSED in 19.0s Stats over 50 runs: max = 19.0s, min = 4.1s, avg = 9.4s, dev = 3.9s Executed 3891 out of 3892 tests: 3891 tests pass and 1 fails to build. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.