==================== Test output for //tensorflow/python/distribute:cross_device_ops_test_2gpu (shard 4 of 4): Running tests under Python 3.11.3: /usr/local/bin/python3 [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 I0514 22:08:38.457150 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: I0514 22:08:38.479895 281472935097216 multi_process_runner.py:840] Subprocess with PID 706078 (worker, 0) is now being started. [worker-0]: I0514 22:08:38.480199 281472935097216 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:19464"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' I0514 22:08:38.717364 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: I0514 22:08:38.820394 281472935097216 multi_process_runner.py:840] Subprocess with PID 707633 (worker, 1) is now being started. [worker-1]: I0514 22:08:38.820728 281472935097216 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:19797", "localhost:18036"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: I0514 22:08:38.837332 281472935097216 multi_process_runner.py:840] Subprocess with PID 707460 (worker, 0) is now being started. [worker-0]: I0514 22:08:38.837663 281472935097216 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:19797", "localhost:18036"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' I0514 22:08:38.867480 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 4.24s I0514 22:08:38.868374 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 4.24s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 I0514 22:08:38.869667 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:38.871461 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:38.883709 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.02s I0514 22:08:38.889200 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.02s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 I0514 22:08:38.890571 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:38.891955 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:38.893178 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s I0514 22:08:38.893629 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 I0514 22:08:38.894685 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:38.895928 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:38.896964 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s I0514 22:08:38.897605 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 I0514 22:08:38.898616 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:38.917073 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:38.918314 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.02s I0514 22:08:38.918783 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.02s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 I0514 22:08:38.919877 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:38.937020 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:38.976272 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.06s I0514 22:08:38.976845 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.06s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 I0514 22:08:38.978038 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:38.979351 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:38.981741 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 I0514 22:08:38.990389 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: E0514 22:08:38.994765461 706078 server_chttp2.cc:40] {"created":"@1684102118.994653834","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1684102118.994649404","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1684102118.994616715","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684102118.994587982","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1684102118.994648618","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684102118.994640398","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]} [worker-0]: 2023-05-14 22:08:38.994875: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:608] UNKNOWN: Could not start gRPC server [ FAILED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.03s I0514 22:08:39.005264 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.03s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 I0514 22:08:39.006571 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: 2023-05-14 22:08:38.995147: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:699] Could not start gRPC server I0514 22:08:39.027593 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:39.028747 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.02s I0514 22:08:39.029234 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.02s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 I0514 22:08:39.030290 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:39.031438 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:39.032354 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s I0514 22:08:39.032755 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 I0514 22:08:39.033734 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:39.034865 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:39.035806 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s I0514 22:08:39.036422 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 I0514 22:08:39.037382 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:39.038521 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:39.039455 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s I0514 22:08:39.039844 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 I0514 22:08:39.040802 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:39.041951 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:39.042888 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s I0514 22:08:39.043292 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 I0514 22:08:39.044255 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:39.045401 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:39.046372 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s I0514 22:08:39.046937 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 I0514 22:08:39.047890 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:39.049028 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:39.049934 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s I0514 22:08:39.050313 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 I0514 22:08:39.051250 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:39.052386 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:39.053268 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s I0514 22:08:39.053637 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 I0514 22:08:39.054565 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:39.055665 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:39.056560 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [worker-0]: E0514 22:08:39.062401896 706078 server_chttp2.cc:40] {"created":"@1684102119.062288420","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1684102119.062283504","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1684102119.062253126","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684102119.062217422","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1684102119.062282829","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684102119.062274868","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]} [worker-0]: 2023-05-14 22:08:39.062489: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:608] UNKNOWN: Could not start gRPC server I0514 22:08:39.062817 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: 2023-05-14 22:08:40.063605: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:699] Could not start gRPC server [ FAILED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 1.02s I0514 22:08:40.069990 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 1.02s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 I0514 22:08:40.071236 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.072475 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.073204 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s I0514 22:08:40.073613 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 I0514 22:08:40.074605 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.075594 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.076044 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s I0514 22:08:40.076500 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 I0514 22:08:40.077488 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.078518 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.078904 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s I0514 22:08:40.079466 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 I0514 22:08:40.080414 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.081422 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.081790 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s I0514 22:08:40.082168 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 I0514 22:08:40.083101 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.084052 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.084419 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s I0514 22:08:40.084799 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 I0514 22:08:40.085733 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.086847 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.087276 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s I0514 22:08:40.087883 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 I0514 22:08:40.088867 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.089908 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.090358 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s I0514 22:08:40.090768 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 I0514 22:08:40.091753 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.092735 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.093104 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s I0514 22:08:40.093483 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 I0514 22:08:40.094426 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.095409 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.095823 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 I0514 22:08:40.100701 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: E0514 22:08:40.103076026 706078 server_chttp2.cc:40] {"created":"@1684102120.102960809","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1684102120.102956853","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1684102120.102925945","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684102120.102895312","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1684102120.102956098","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684102120.102948223","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]} [worker-0]: 2023-05-14 22:08:40.103163: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:608] UNKNOWN: Could not start gRPC server [worker-0]: 2023-05-14 22:08:40.116501: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:699] Could not start gRPC server [ FAILED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.03s I0514 22:08:40.122684 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.03s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 I0514 22:08:40.123933 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.125185 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.125840 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.01s I0514 22:08:40.129806 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.01s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 I0514 22:08:40.131030 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.132227 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.133878 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s I0514 22:08:40.134329 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 I0514 22:08:40.135407 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.136509 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.136936 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s I0514 22:08:40.137520 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 I0514 22:08:40.138505 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.139548 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.140137 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s I0514 22:08:40.140523 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 I0514 22:08:40.141490 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.142539 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.142898 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s I0514 22:08:40.143288 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 I0514 22:08:40.144252 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.145297 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.145698 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s I0514 22:08:40.146325 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 I0514 22:08:40.147295 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.148344 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.148736 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s I0514 22:08:40.149115 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 I0514 22:08:40.150061 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.151142 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.151727 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s I0514 22:08:40.152374 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 I0514 22:08:40.153376 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.154409 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.154799 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 I0514 22:08:40.159285 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: E0514 22:08:40.161054299 706078 server_chttp2.cc:40] {"created":"@1684102120.160941692","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1684102120.160937037","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1684102120.160906674","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684102120.160874320","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1684102120.160936137","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684102120.160928546","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]} [worker-0]: 2023-05-14 22:08:40.161141: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:608] UNKNOWN: Could not start gRPC server [worker-0]: 2023-05-14 22:08:40.161414: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:699] Could not start gRPC server [ FAILED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.01s I0514 22:08:40.167510 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.01s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 I0514 22:08:40.168756 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.169958 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.170618 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s I0514 22:08:40.171340 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 I0514 22:08:40.172394 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.173558 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.174173 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s I0514 22:08:40.174587 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 I0514 22:08:40.175578 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.176640 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.177074 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s I0514 22:08:40.177653 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 I0514 22:08:40.178622 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.179695 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.180098 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s I0514 22:08:40.180492 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 I0514 22:08:40.181460 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.182476 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.182924 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s I0514 22:08:40.183319 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 I0514 22:08:40.184272 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.185269 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.185647 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s I0514 22:08:40.186213 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 I0514 22:08:40.187162 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.188126 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.188523 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s I0514 22:08:40.188897 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 I0514 22:08:40.189820 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.190786 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.191189 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s I0514 22:08:40.191598 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 I0514 22:08:40.192570 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.193649 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.194037 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 I0514 22:08:40.198972 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: E0514 22:08:40.200722427 706078 server_chttp2.cc:40] {"created":"@1684102120.200608510","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1684102120.200604070","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1684102120.200573016","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684102120.200548124","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1684102120.200603294","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684102120.200595074","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]} [worker-0]: 2023-05-14 22:08:40.200815: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:608] UNKNOWN: Could not start gRPC server [worker-0]: 2023-05-14 22:08:40.201078: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:699] Could not start gRPC server [ FAILED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.02s I0514 22:08:40.207392 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.02s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 I0514 22:08:40.208637 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.209957 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.210420 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s I0514 22:08:40.210851 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 I0514 22:08:40.211891 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.212928 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.213342 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s I0514 22:08:40.213752 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 I0514 22:08:40.214728 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.215762 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.216165 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s I0514 22:08:40.216726 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 I0514 22:08:40.217675 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.218745 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.219163 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s I0514 22:08:40.219583 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 I0514 22:08:40.220582 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.221721 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.227363 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.01s I0514 22:08:40.227904 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.01s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 I0514 22:08:40.229063 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.230216 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.230648 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s I0514 22:08:40.231239 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 I0514 22:08:40.232248 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.233279 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.233705 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s I0514 22:08:40.234107 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 I0514 22:08:40.235087 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.236293 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.236651 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.02s I0514 22:08:40.256633 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.02s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 I0514 22:08:40.257870 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.259282 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.260349 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 I0514 22:08:40.264894 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: E0514 22:08:40.266877499 706078 server_chttp2.cc:40] {"created":"@1684102120.266761712","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1684102120.266757471","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1684102120.266726328","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684102120.266703286","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1684102120.266756731","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684102120.266748240","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]} [worker-0]: 2023-05-14 22:08:40.266963: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:608] UNKNOWN: Could not start gRPC server [worker-0]: 2023-05-14 22:08:40.267737: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:699] Could not start gRPC server [ FAILED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.02s I0514 22:08:40.274184 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0): 0.02s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 I0514 22:08:40.275458 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.276657 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.277390 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s I0514 22:08:40.277807 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 I0514 22:08:40.278818 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.279969 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.280381 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s I0514 22:08:40.280790 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_reduceop_ReduceOpSUM_requiredgpus_0 I0514 22:08:40.281793 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.282816 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.283255 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_reduceop_ReduceOpSUM_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_reduceop_ReduceOpSUM_requiredgpus_0): 0.0s I0514 22:08:40.283840 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_reduceop_ReduceOpSUM_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_1 I0514 22:08:40.284838 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.285926 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.286420 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_1): 0.0s I0514 22:08:40.286827 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_2 I0514 22:08:40.287808 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.288866 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.289300 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_2): 0.0s I0514 22:08:40.289718 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_reduceop_ReduceOpSUM_requiredgpus_0 I0514 22:08:40.290714 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.291822 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.292423 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_reduceop_ReduceOpSUM_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_reduceop_ReduceOpSUM_requiredgpus_0): 0.0s I0514 22:08:40.293004 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_reduceop_ReduceOpSUM_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_1 I0514 22:08:40.293990 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.295008 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.295454 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_1): 0.0s I0514 22:08:40.295842 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_2 I0514 22:08:40.296855 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.297947 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.298359 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_2): 0.0s I0514 22:08:40.298794 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_reduceop_ReduceOpSUM_requiredgpus_0 I0514 22:08:40.299771 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.300940 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.301354 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 I0514 22:08:40.306257 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: E0514 22:08:40.308213810 706078 server_chttp2.cc:40] {"created":"@1684102120.308101038","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1684102120.308096748","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1684102120.308065464","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684102120.308037671","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1684102120.308095727","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684102120.308087687","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]} [worker-0]: 2023-05-14 22:08:40.308302: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:608] UNKNOWN: Could not start gRPC server [worker-0]: 2023-05-14 22:08:40.308686: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:699] Could not start gRPC server [ FAILED ] CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_reduceop_ReduceOpSUM_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_reduceop_ReduceOpSUM_requiredgpus_0): 0.02s I0514 22:08:40.314910 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_reduceop_ReduceOpSUM_requiredgpus_0): 0.02s [ RUN ] CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_1 I0514 22:08:40.316205 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.317434 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.317911 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_1): 0.0s I0514 22:08:40.318319 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_2 I0514 22:08:40.319337 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.320394 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.320848 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_2): 0.0s I0514 22:08:40.321246 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_1_reduceop_ReduceOpMEAN_requiredgpus_2 I0514 22:08:40.322259 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.323298 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.323720 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_1_reduceop_ReduceOpMEAN_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_1_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s I0514 22:08:40.324131 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_1_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_0 I0514 22:08:40.325133 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.326195 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.326688 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_0): 0.0s I0514 22:08:40.327269 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:40.328267 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.329489 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.329950 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s I0514 22:08:40.330376 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_1_reduceop_ReduceOpMEAN_requiredgpus_2 I0514 22:08:40.331379 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.332497 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.332928 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_1_reduceop_ReduceOpMEAN_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_1_reduceop_ReduceOpMEAN_requiredgpus_2): 0.02s I0514 22:08:40.347100 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_1_reduceop_ReduceOpMEAN_requiredgpus_2): 0.02s [ RUN ] CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_0 I0514 22:08:40.348341 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.349708 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.350935 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_0): 0.0s I0514 22:08:40.351556 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:40.352660 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.353966 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.354582 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s I0514 22:08:40.354996 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_1_reduceop_ReduceOpMEAN_requiredgpus_2 I0514 22:08:40.356019 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.357272 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.358278 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_1_reduceop_ReduceOpMEAN_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_1_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s I0514 22:08:40.358703 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_1_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_0 I0514 22:08:40.359749 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.360991 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:40.361998 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 I0514 22:08:40.366610 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: 2023-05-14 22:08:40.450706: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:18036 [worker-0]: 2023-05-14 22:08:40.517638: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:19797 [worker-1]: INFO:tensorflow:Collective all_reduce IndexedSlices: 1 all_reduces, num_devices =1, group_size = 2, implementation = CommunicationImplementation.RING [worker-1]: I0514 22:08:40.669078 281472935097216 cross_device_ops.py:1167] Collective all_reduce IndexedSlices: 1 all_reduces, num_devices =1, group_size = 2, implementation = CommunicationImplementation.RING [worker-0]: INFO:tensorflow:Collective all_reduce IndexedSlices: 1 all_reduces, num_devices =1, group_size = 2, implementation = CommunicationImplementation.RING [worker-0]: I0514 22:08:40.794468 281472935097216 cross_device_ops.py:1167] Collective all_reduce IndexedSlices: 1 all_reduces, num_devices =1, group_size = 2, implementation = CommunicationImplementation.RING [worker-1]: INFO:tensorflow:Collective all_reduce IndexedSlices: 2 all_reduces, num_devices =1, group_size = 2, implementation = CommunicationImplementation.RING [worker-1]: I0514 22:08:41.485412 281472935097216 cross_device_ops.py:1167] Collective all_reduce IndexedSlices: 2 all_reduces, num_devices =1, group_size = 2, implementation = CommunicationImplementation.RING [worker-0]: INFO:tensorflow:Collective all_reduce IndexedSlices: 2 all_reduces, num_devices =1, group_size = 2, implementation = CommunicationImplementation.RING [worker-0]: I0514 22:08:41.527017 281472935097216 cross_device_ops.py:1167] Collective all_reduce IndexedSlices: 2 all_reduces, num_devices =1, group_size = 2, implementation = CommunicationImplementation.RING I0514 22:08:42.207808 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_0): 1.85s I0514 22:08:42.208147 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_0): 1.85s [ OK ] CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_reduceop_ReduceOpMEAN_requiredgpus_0 [ RUN ] CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:42.209404 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:42.210700 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:42.211277 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:42.213809 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:42.216291 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_1): 0.01s I0514 22:08:42.216760 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testAllReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_reduceop_ReduceOpSUM_requiredgpus_1): 0.01s [ RUN ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 I0514 22:08:42.217843 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:42.219039 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:42.219533 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:42.220533 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:42.221134 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s I0514 22:08:42.221534 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 I0514 22:08:42.222508 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:42.223640 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:42.224085 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:42.224967 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:42.225550 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.0s I0514 22:08:42.226131 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:42.227111 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:42.228261 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:42.228717 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:42.229620 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:42.230209 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s I0514 22:08:42.230599 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 I0514 22:08:42.231567 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:42.232708 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:42.233147 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:42.233980 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:42.234528 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s I0514 22:08:42.234913 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 I0514 22:08:42.235879 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:42.237050 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:42.237492 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:42.238319 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:42.238857 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.0s I0514 22:08:42.239398 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:42.240350 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:42.241476 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:42.241890 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:42.242746 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:42.243286 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s I0514 22:08:42.243661 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 I0514 22:08:42.244603 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:42.245722 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:42.247233 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:42.256586 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:42.257341 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.01s I0514 22:08:42.257829 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.01s [ RUN ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 I0514 22:08:42.258972 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:42.260203 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:42.260706 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:42.261667 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:42.262248 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.0s I0514 22:08:42.262830 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:42.263859 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:42.265048 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:42.265524 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:42.266430 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:42.310715 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.05s I0514 22:08:42.311225 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.05s [ RUN ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 I0514 22:08:42.312370 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:42.313645 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:42.314153 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:42.315170 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:42.315759 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s I0514 22:08:42.316177 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 I0514 22:08:42.317194 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:42.318361 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:42.318810 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:42.319795 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:42.320363 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.0s I0514 22:08:42.320927 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:42.321903 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:42.322926 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:42.323197 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:42.323368 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:42.323976 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s I0514 22:08:42.324362 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 I0514 22:08:42.325309 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:42.326335 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:42.326785 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:42.327650 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:42.328229 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s I0514 22:08:42.328650 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 I0514 22:08:42.329718 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:42.331045 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:42.331346 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:42.331530 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:42.349398 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [worker-0]: E0514 22:08:42.356920140 706078 server_chttp2.cc:40] {"created":"@1684102122.356798733","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1684102122.356794417","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1684102122.356763869","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684102122.356733636","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1684102122.356793777","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684102122.356786056","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]} [worker-0]: 2023-05-14 22:08:42.357014: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:608] UNKNOWN: Could not start gRPC server I0514 22:08:42.357423 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: 2023-05-14 22:08:42.386594: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:699] Could not start gRPC server [ FAILED ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.06s I0514 22:08:42.393396 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.06s [ RUN ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:42.394653 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:42.417109 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:42.417711 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:42.418694 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:42.419368 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.03s I0514 22:08:42.419836 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.03s [ RUN ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 I0514 22:08:42.420940 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:42.422153 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:42.422595 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:42.423465 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:42.424027 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s I0514 22:08:42.424423 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 I0514 22:08:42.425409 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:42.426582 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:42.427039 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:42.427885 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:42.428435 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 I0514 22:08:42.433283 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 2 all_reduces, num_devices = 1, group_size = 2, implementation = CommunicationImplementation.RING, num_packs = 1 [worker-0]: I0514 22:08:43.315568 281472935097216 cross_device_ops.py:1152] Collective all_reduce tensors: 2 all_reduces, num_devices = 1, group_size = 2, implementation = CommunicationImplementation.RING, num_packs = 1 [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py:1554: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: tf.py_func is deprecated in TF V2. Instead, there are two [worker-0]: options available in V2. [worker-0]: - tf.py_function takes a python function which manipulates tf eager [worker-0]: tensors instead of numpy arrays. It's easy to convert a tf eager tensor to [worker-0]: an ndarray (just call tensor.numpy()) but having access to eager tensors [worker-0]: means `tf.py_function`s can use accelerators such as GPUs as well as [worker-0]: being differentiable using a gradient tape. [worker-0]: - tf.numpy_function maintains the semantics of the deprecated tf.py_func [worker-0]: (it is not differentiable, and manipulates numpy arrays). It drops the [worker-0]: stateful argument making all functions stateful. [worker-0]: [worker-0]: W0514 22:08:43.341839 281472935097216 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py:1554: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: tf.py_func is deprecated in TF V2. Instead, there are two [worker-0]: options available in V2. [worker-0]: - tf.py_function takes a python function which manipulates tf eager [worker-0]: tensors instead of numpy arrays. It's easy to convert a tf eager tensor to [worker-0]: an ndarray (just call tensor.numpy()) but having access to eager tensors [worker-0]: means `tf.py_function`s can use accelerators such as GPUs as well as [worker-0]: being differentiable using a gradient tape. [worker-0]: - tf.numpy_function maintains the semantics of the deprecated tf.py_func [worker-0]: (it is not differentiable, and manipulates numpy arrays). It drops the [worker-0]: stateful argument making all functions stateful. [worker-0]: [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 2 all_reduces, num_devices = 1, group_size = 2, implementation = CommunicationImplementation.RING, num_packs = 1 [worker-1]: I0514 22:08:43.377320 281472935097216 cross_device_ops.py:1152] Collective all_reduce tensors: 2 all_reduces, num_devices = 1, group_size = 2, implementation = CommunicationImplementation.RING, num_packs = 1 [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py:1554: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: tf.py_func is deprecated in TF V2. Instead, there are two [worker-1]: options available in V2. [worker-1]: - tf.py_function takes a python function which manipulates tf eager [worker-1]: tensors instead of numpy arrays. It's easy to convert a tf eager tensor to [worker-1]: an ndarray (just call tensor.numpy()) but having access to eager tensors [worker-1]: means `tf.py_function`s can use accelerators such as GPUs as well as [worker-1]: being differentiable using a gradient tape. [worker-1]: - tf.numpy_function maintains the semantics of the deprecated tf.py_func [worker-1]: (it is not differentiable, and manipulates numpy arrays). It drops the [worker-1]: stateful argument making all functions stateful. [worker-1]: [worker-1]: W0514 22:08:43.403069 281472935097216 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py:1554: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: tf.py_func is deprecated in TF V2. Instead, there are two [worker-1]: options available in V2. [worker-1]: - tf.py_function takes a python function which manipulates tf eager [worker-1]: tensors instead of numpy arrays. It's easy to convert a tf eager tensor to [worker-1]: an ndarray (just call tensor.numpy()) but having access to eager tensors [worker-1]: means `tf.py_function`s can use accelerators such as GPUs as well as [worker-1]: being differentiable using a gradient tape. [worker-1]: - tf.numpy_function maintains the semantics of the deprecated tf.py_func [worker-1]: (it is not differentiable, and manipulates numpy arrays). It drops the [worker-1]: stateful argument making all functions stateful. [worker-1]: I0514 22:08:43.448724 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 1.02s I0514 22:08:43.449196 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 1.02s [ OK ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 [ RUN ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:43.450493 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:43.453880 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:43.454524 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:43.456455 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [worker-1]: W0514 22:08:43.476758 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [ SKIPPED ] CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.15s I0514 22:08:43.597546 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.15s [ RUN ] CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:43.598778 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:43.600055 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:43.600556 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:43.601275 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [worker-1]: W0514 22:08:43.601768 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [ SKIPPED ] CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s I0514 22:08:43.602622 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_2 I0514 22:08:43.603620 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:43.604780 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:43.605224 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:43.605795 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [worker-1]: W0514 22:08:43.606260 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [ SKIPPED ] CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_2): 0.0s I0514 22:08:43.607090 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_0 I0514 22:08:43.608076 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:43.609228 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:43.609674 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:43.610237 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [worker-1]: W0514 22:08:43.610675 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [ SKIPPED ] CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_0): 0.0s I0514 22:08:43.611635 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:43.612596 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:43.613729 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:43.614149 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:43.614676 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [worker-1]: W0514 22:08:43.615086 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [ SKIPPED ] CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s I0514 22:08:43.615806 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_2 I0514 22:08:43.616752 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:43.617855 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:43.618256 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:43.618746 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [worker-1]: W0514 22:08:43.619141 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [ SKIPPED ] CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_2): 0.0s I0514 22:08:43.619840 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_0 I0514 22:08:43.620761 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:43.621848 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:43.622241 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:43.622718 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [worker-1]: W0514 22:08:43.623105 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [ SKIPPED ] CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_0): 0.0s I0514 22:08:43.623939 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:43.624843 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:43.625928 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:43.626394 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:43.626888 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [worker-1]: W0514 22:08:43.627282 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [ SKIPPED ] CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s I0514 22:08:43.627973 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_2 I0514 22:08:43.628895 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:43.629979 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:43.630376 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:43.631086 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:43.631577 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_2): 0.0s I0514 22:08:43.631929 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_0 I0514 22:08:43.632863 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:43.647118 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:43.647726 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:43.648773 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:43.650166 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 I0514 22:08:43.655586 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: INFO:tensorflow:Collective all_reduce IndexedSlices: 2 all_reduces, num_devices =1, group_size = 2, implementation = CommunicationImplementation.RING [worker-1]: I0514 22:08:43.703452 281472935097216 cross_device_ops.py:1167] Collective all_reduce IndexedSlices: 2 all_reduces, num_devices =1, group_size = 2, implementation = CommunicationImplementation.RING [worker-0]: INFO:tensorflow:Collective all_reduce IndexedSlices: 2 all_reduces, num_devices =1, group_size = 2, implementation = CommunicationImplementation.RING [worker-0]: I0514 22:08:43.709613 281472935097216 cross_device_ops.py:1167] Collective all_reduce IndexedSlices: 2 all_reduces, num_devices =1, group_size = 2, implementation = CommunicationImplementation.RING [worker-0]: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py:2988: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray. [worker-0]: return np.array(a) I0514 22:08:44.913648 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [worker-1]: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py:2988: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray. [worker-1]: return np.array(a) INFO:tensorflow:time(__main__.CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_0): 1.28s I0514 22:08:44.916425 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_0): 1.28s [ OK ] CollectiveOpsTest.testBatchReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_0 [ RUN ] CollectiveOpsTest.testCollectiveV2ControlFlow_test_implementation_CommunicationImplementationRING_numprocesses_1_requiredgpus_1 I0514 22:08:44.917772 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:44.918970 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:44.919358 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:44.919387 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:44.920672 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testCollectiveV2ControlFlow_test_implementation_CommunicationImplementationRING_numprocesses_1_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testCollectiveV2ControlFlow_test_implementation_CommunicationImplementationRING_numprocesses_1_requiredgpus_1): 0.0s I0514 22:08:44.921117 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testCollectiveV2ControlFlow_test_implementation_CommunicationImplementationRING_numprocesses_1_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testCollectiveV2ControlFlow_test_implementation_CommunicationImplementationRING_numprocesses_2_requiredgpus_2 I0514 22:08:44.922199 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:44.923290 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:44.923577 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:44.923642 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:44.924181 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testCollectiveV2ControlFlow_test_implementation_CommunicationImplementationRING_numprocesses_2_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testCollectiveV2ControlFlow_test_implementation_CommunicationImplementationRING_numprocesses_2_requiredgpus_2): 0.0s I0514 22:08:44.924573 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testCollectiveV2ControlFlow_test_implementation_CommunicationImplementationRING_numprocesses_2_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testInputsAreFunctionArgs_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_2 I0514 22:08:44.925557 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:44.926610 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:44.926910 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:44.926962 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:44.927495 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testInputsAreFunctionArgs_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testInputsAreFunctionArgs_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_2): 0.0s I0514 22:08:44.927902 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testInputsAreFunctionArgs_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testMultiThreadedCollectiveLaunchNoInterleave_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_2 I0514 22:08:44.928895 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:44.929944 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:44.930238 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:44.930279 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:44.930790 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testMultiThreadedCollectiveLaunchNoInterleave_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testMultiThreadedCollectiveLaunchNoInterleave_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_2): 0.0s I0514 22:08:44.931188 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testMultiThreadedCollectiveLaunchNoInterleave_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 I0514 22:08:44.932152 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:44.933170 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:44.933442 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:44.933528 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:44.933960 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s I0514 22:08:44.934348 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 I0514 22:08:44.935297 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:44.936375 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:44.936697 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:44.936865 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:44.937367 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.0s I0514 22:08:44.937940 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:44.938921 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:44.940014 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:44.940355 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:44.940362 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:44.940968 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s I0514 22:08:44.941391 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 I0514 22:08:44.942413 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:44.943532 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:44.943861 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:44.944533 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s I0514 22:08:44.944995 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 I0514 22:08:44.946065 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:44.944042 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:44.951070 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:44.951869 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:44.951498 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:44.952738 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.01s I0514 22:08:44.953492 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.01s [ RUN ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:44.954704 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:44.956044 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:44.956463 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:44.956522 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:44.957292 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s I0514 22:08:44.957746 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 I0514 22:08:44.958833 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:44.960046 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:44.960444 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:44.960562 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:44.961223 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s I0514 22:08:44.961653 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 I0514 22:08:44.962692 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:44.963824 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:44.964200 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:44.964218 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:44.964874 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.0s I0514 22:08:44.965502 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:44.966600 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:44.967808 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:44.968378 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:44.969153 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [worker-0]: W0514 22:08:44.968239 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [ SKIPPED ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s I0514 22:08:44.969641 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 I0514 22:08:44.970708 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:44.971914 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:44.972312 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:44.972460 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:44.973177 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s I0514 22:08:44.973633 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 I0514 22:08:44.974705 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:44.977519 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:44.977975 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:44.978172 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:44.978797 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.01s I0514 22:08:44.979465 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.01s [ RUN ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:44.980552 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:44.981941 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:44.982319 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:44.982374 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:44.983031 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s I0514 22:08:44.983471 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 I0514 22:08:44.984520 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:44.985686 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:44.986253 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:44.987418 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:44.991131 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.01s I0514 22:08:44.991761 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.01s [ RUN ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 I0514 22:08:44.993090 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 I0514 22:08:44.995507 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:44.996187 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:44.997612 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:44.998388 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 I0514 22:08:45.003400 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: 2023-05-14 22:08:45.017201: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:19464 [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py:1554: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: tf.py_func is deprecated in TF V2. Instead, there are two [worker-0]: options available in V2. [worker-0]: - tf.py_function takes a python function which manipulates tf eager [worker-0]: tensors instead of numpy arrays. It's easy to convert a tf eager tensor to [worker-0]: an ndarray (just call tensor.numpy()) but having access to eager tensors [worker-0]: means `tf.py_function`s can use accelerators such as GPUs as well as [worker-0]: being differentiable using a gradient tape. [worker-0]: - tf.numpy_function maintains the semantics of the deprecated tf.py_func [worker-0]: (it is not differentiable, and manipulates numpy arrays). It drops the [worker-0]: stateful argument making all functions stateful. [worker-0]: [worker-0]: W0514 22:08:45.922923 281472935097216 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py:1554: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: tf.py_func is deprecated in TF V2. Instead, there are two [worker-0]: options available in V2. [worker-0]: - tf.py_function takes a python function which manipulates tf eager [worker-0]: tensors instead of numpy arrays. It's easy to convert a tf eager tensor to [worker-0]: an ndarray (just call tensor.numpy()) but having access to eager tensors [worker-0]: means `tf.py_function`s can use accelerators such as GPUs as well as [worker-0]: being differentiable using a gradient tape. [worker-0]: - tf.numpy_function maintains the semantics of the deprecated tf.py_func [worker-0]: (it is not differentiable, and manipulates numpy arrays). It drops the [worker-0]: stateful argument making all functions stateful. [worker-0]: INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.95s I0514 22:08:45.941542 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.95s [ OK ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 [ RUN ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:45.943107 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:45.943837 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:45.945627 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:45.946346 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:45.947536 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:45.948278 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.01s I0514 22:08:45.948783 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.01s [ RUN ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 I0514 22:08:45.949985 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:45.950628 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:45.952131 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:45.952776 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:45.953912 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:45.954621 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.01s I0514 22:08:45.955102 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpMEAN_requiredgpus_2): 0.01s [ RUN ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 I0514 22:08:45.956307 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:45.956958 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:45.958377 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:45.958954 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:45.960113 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:45.960788 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 I0514 22:08:45.965766 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 2, implementation = CommunicationImplementation.RING, num_packs = 1 [worker-0]: I0514 22:08:46.170547 281472935097216 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 2, implementation = CommunicationImplementation.RING, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 2, implementation = CommunicationImplementation.RING, num_packs = 1 [worker-1]: I0514 22:08:46.246412 281472935097216 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 2, implementation = CommunicationImplementation.RING, num_packs = 1 I0514 22:08:46.314995 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.36s I0514 22:08:46.315412 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0): 0.36s [ OK ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 [ RUN ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:46.317023 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:46.317548 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.319286 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:46.319830 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:46.319991 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.331772 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.02s I0514 22:08:46.332317 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.02s [ RUN ] CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_0 I0514 22:08:46.333518 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:46.334092 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.335788 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:46.336373 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:46.339849 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.340550 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_0): 0.01s I0514 22:08:46.341217 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_0): 0.01s [ RUN ] CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:46.342402 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:46.342972 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.344422 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:46.344974 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:46.346006 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.346690 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.01s I0514 22:08:46.347160 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.01s [ RUN ] CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_2 I0514 22:08:46.348323 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:46.348866 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.367105 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:46.367723 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:46.368396 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.368980 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_2): 0.02s I0514 22:08:46.369508 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationAUTO_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_2): 0.02s [ RUN ] CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_0 I0514 22:08:46.370704 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:46.371115 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.372469 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:46.372895 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.375950 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_0): 0.01s I0514 22:08:46.376691 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_0): 0.01s [ RUN ] CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:46.377857 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:46.375231 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:46.378459 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.379872 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:46.380223 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:46.380270 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.380839 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s I0514 22:08:46.381270 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_2 I0514 22:08:46.382350 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:46.382700 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.383933 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:46.384255 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.384835 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [worker-1]: W0514 22:08:46.384427 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [ SKIPPED ] CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_2): 0.0s I0514 22:08:46.385336 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_0 I0514 22:08:46.386487 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:46.386862 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.388473 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:46.388797 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:46.388860 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.389385 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 I0514 22:08:46.394543 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_0): 0.08s I0514 22:08:46.462549 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_0): 0.08s [ OK ] CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_0 [ RUN ] CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 I0514 22:08:46.463990 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:46.464415 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.466175 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:46.466592 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:46.466744 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.467313 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s I0514 22:08:46.467814 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpSUM_requiredgpus_1): 0.0s [ RUN ] CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_2 I0514 22:08:46.468958 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:46.469622 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.471074 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:46.471411 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:46.471480 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.472029 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_2 INFO:tensorflow:time(__main__.CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_2): 0.0s I0514 22:08:46.472482 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_reduceop_ReduceOpSUM_requiredgpus_2): 0.0s [ RUN ] CollectiveOpsTest.testTimeoutBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0 I0514 22:08:46.473588 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:46.473988 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.475309 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:46.475637 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:46.475712 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.476271 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testTimeoutBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testTimeoutBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0): 0.0s I0514 22:08:46.477061 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testTimeoutBatchReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0): 0.0s [ RUN ] CollectiveOpsTest.testTimeoutBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0 I0514 22:08:46.478196 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:46.478612 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.479997 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:46.480327 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-1]: W0514 22:08:46.480380 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:46.480930 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 I0514 22:08:46.485248 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 2 all_reduces, num_devices = 1, group_size = 2, implementation = CommunicationImplementation.RING, num_packs = 1 [worker-0]: I0514 22:08:46.537453 281472935097216 cross_device_ops.py:1152] Collective all_reduce tensors: 2 all_reduces, num_devices = 1, group_size = 2, implementation = CommunicationImplementation.RING, num_packs = 1 [worker-0]: 2023-05-14 22:08:47.606190: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort DEADLINE_EXCEEDED: Collective has timed out waiting for other workers. [worker-0]: 2023-05-14 22:08:47.606277: I tensorflow/core/common_runtime/executor.cc:1210] [/job:worker/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): DEADLINE_EXCEEDED: Collective has timed out waiting for other workers. [worker-0]: [[{{node CollectiveReduceV2}}]] I0514 22:08:47.620109 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testTimeoutBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0): 1.14s I0514 22:08:47.620466 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testTimeoutBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0): 1.14s [ OK ] CollectiveOpsTest.testTimeoutBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0 [ RUN ] CollectiveOpsTest.testTimeoutBatchReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0 I0514 22:08:47.621743 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:47.622289 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:47.623637 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:47.624159 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:47.625127 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:47.625992 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testTimeoutBatchReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testTimeoutBatchReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0): 0.01s I0514 22:08:47.626614 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testTimeoutBatchReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0): 0.01s [ RUN ] CollectiveOpsTest.testTimeoutBatchReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0 I0514 22:08:47.627668 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:47.628160 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:47.629371 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:47.629836 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:47.630728 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:47.631285 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 I0514 22:08:47.634901 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: INFO:tensorflow:Collective all_reduce IndexedSlices: 2 all_reduces, num_devices =1, group_size = 2, implementation = CommunicationImplementation.RING [worker-0]: I0514 22:08:47.708594 281472935097216 cross_device_ops.py:1167] Collective all_reduce IndexedSlices: 2 all_reduces, num_devices =1, group_size = 2, implementation = CommunicationImplementation.RING [worker-0]: 2023-05-14 22:08:49.318050: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort DEADLINE_EXCEEDED: Collective has timed out waiting for other workers. [worker-0]: 2023-05-14 22:08:49.318130: I tensorflow/core/common_runtime/executor.cc:1210] [/job:worker/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): DEADLINE_EXCEEDED: Collective has timed out waiting for other workers. [worker-0]: [[{{node CollectiveGatherV2}}]] I0514 22:08:49.325667 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testTimeoutBatchReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0): 1.7s I0514 22:08:49.325983 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testTimeoutBatchReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0): 1.7s [ OK ] CollectiveOpsTest.testTimeoutBatchReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0 [ RUN ] CollectiveOpsTest.testTimeoutReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0 I0514 22:08:49.327317 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:49.327883 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:49.329352 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:49.329912 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:49.330968 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:49.341589 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testTimeoutReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testTimeoutReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0): 0.02s I0514 22:08:49.342304 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testTimeoutReduceDense_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0): 0.02s [ RUN ] CollectiveOpsTest.testTimeoutReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0 I0514 22:08:49.343483 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:49.344035 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:49.345529 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:49.346053 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:49.347114 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:49.347737 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 I0514 22:08:49.351894 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 2, implementation = CommunicationImplementation.RING, num_packs = 1 [worker-0]: I0514 22:08:49.419141 281472935097216 cross_device_ops.py:1152] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 2, implementation = CommunicationImplementation.RING, num_packs = 1 [worker-0]: 2023-05-14 22:08:50.480646: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort DEADLINE_EXCEEDED: Collective has timed out waiting for other workers. [worker-0]: 2023-05-14 22:08:50.480736: I tensorflow/core/common_runtime/executor.cc:1210] [/job:worker/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): DEADLINE_EXCEEDED: Collective has timed out waiting for other workers. [worker-0]: [[{{node CollectiveReduceV2}}]] I0514 22:08:50.484298 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testTimeoutReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0): 1.14s I0514 22:08:50.484681 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testTimeoutReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0): 1.14s [ OK ] CollectiveOpsTest.testTimeoutReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0 [ RUN ] CollectiveOpsTest.testTimeoutReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0 I0514 22:08:50.486107 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:50.486856 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:50.488469 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:50.489119 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:50.490376 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:50.491420 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 [ SKIPPED ] CollectiveOpsTest.testTimeoutReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0 INFO:tensorflow:time(__main__.CollectiveOpsTest.testTimeoutReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0): 0.01s I0514 22:08:50.492095 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testTimeoutReduceSparse_test_implementation_CommunicationImplementationNCCL_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0): 0.01s [ RUN ] CollectiveOpsTest.testTimeoutReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0 I0514 22:08:50.493240 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: W0514 22:08:50.493904 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:50.495218 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-1]: W0514 22:08:50.495766 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. [worker-0]: W0514 22:08:50.496887 281472935097216 context.py:881] Enabling collective ops after program startup may cause error when accessing previously created tensors. I0514 22:08:50.498438 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 I0514 22:08:50.502289 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-0 [worker-0]: INFO:tensorflow:Collective all_reduce IndexedSlices: 1 all_reduces, num_devices =1, group_size = 2, implementation = CommunicationImplementation.RING [worker-0]: I0514 22:08:50.564344 281472935097216 cross_device_ops.py:1167] Collective all_reduce IndexedSlices: 1 all_reduces, num_devices =1, group_size = 2, implementation = CommunicationImplementation.RING [worker-0]: 2023-05-14 22:08:51.841369: E tensorflow/core/common_runtime/base_collective_executor.cc:249] BaseCollectiveExecutor::StartAbort DEADLINE_EXCEEDED: Collective has timed out waiting for other workers. [worker-0]: 2023-05-14 22:08:51.841471: I tensorflow/core/common_runtime/executor.cc:1210] [/job:worker/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): DEADLINE_EXCEEDED: Collective has timed out waiting for other workers. [worker-0]: [[{{node CollectiveGatherV2}}]] I0514 22:08:51.857531 281473285059456 multi_process_runner.py:989] Waiting for the result from worker-1 INFO:tensorflow:time(__main__.CollectiveOpsTest.testTimeoutReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0): 1.37s I0514 22:08:51.857938 281473285059456 test_util.py:2464] time(__main__.CollectiveOpsTest.testTimeoutReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0): 1.37s [ OK ] CollectiveOpsTest.testTimeoutReduceSparse_test_implementation_CommunicationImplementationRING_numprocesses_2_preferuniqueinstancekey_False_requiredgpus_0 [ RUN ] CollectiveOpsTest.test_session [ SKIPPED ] CollectiveOpsTest.test_session I0514 22:08:52.756538 281473285059456 multi_process_runner.py:646] worker-0 exit code: 0 I0514 22:08:52.756742 281473285059456 multi_process_runner.py:646] worker-1 exit code: 0 I0514 22:08:52.758976 281473285059456 multi_process_runner.py:662] Joining log reading threads. I0514 22:08:52.759152 281473285059456 multi_process_runner.py:665] Joined log reading threads. I0514 22:08:53.490753 281473285059456 multi_process_runner.py:646] worker-0 exit code: 0 I0514 22:08:53.492094 281473285059456 multi_process_runner.py:662] Joining log reading threads. I0514 22:08:53.492271 281473285059456 multi_process_runner.py:665] Joined log reading threads. ====================================================================== ERROR: testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 (__main__.CollectiveOpsTest) CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 testAllGatherSameShape_test_axis_0_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0(axis=0, func_mode='eager', implementation=, num_processes=1, prefer_unique_instance_key=True, required_gpus=0) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 889, in testAllGatherSameShape get_global_mpr(num_processes).run(replica_fn) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1003, in run six.reraise(*process_status.exc_info) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/six_archive/six.py", line 719, in reraise raise value File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained return_value = fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 861, in replica_fn collective, devices, _ = self.make_collective(num_processes, ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 185, in make_collective collective = cross_device_ops_lib.CollectiveAllReduce( ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops.py", line 1089, in __init__ self._devices = tuple(device_util.canonicalize(d) for d in devices) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops.py", line 1089, in self._devices = tuple(device_util.canonicalize(d) for d in devices) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/device_util.py", line 60, in canonicalize config.list_logical_devices("CPU")[0].name) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/config.py", line 491, in list_logical_devices return context.context().list_logical_devices(device_type=device_type) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 1688, in list_logical_devices self.ensure_initialized() ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 610, in ensure_initialized pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) ^^^^^^^^^^^^^^^^^ tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server ====================================================================== ERROR: testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 (__main__.CollectiveOpsTest) CollectiveOpsTest.testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 testAllGatherSameShape_test_axis_0_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0(axis=0, func_mode='func_graph', implementation=, num_processes=1, prefer_unique_instance_key=True, required_gpus=0) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 889, in testAllGatherSameShape get_global_mpr(num_processes).run(replica_fn) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1003, in run six.reraise(*process_status.exc_info) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/six_archive/six.py", line 719, in reraise raise value File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained return_value = fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 861, in replica_fn collective, devices, _ = self.make_collective(num_processes, ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 185, in make_collective collective = cross_device_ops_lib.CollectiveAllReduce( ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops.py", line 1089, in __init__ self._devices = tuple(device_util.canonicalize(d) for d in devices) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops.py", line 1089, in self._devices = tuple(device_util.canonicalize(d) for d in devices) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/device_util.py", line 60, in canonicalize config.list_logical_devices("CPU")[0].name) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/config.py", line 491, in list_logical_devices return context.context().list_logical_devices(device_type=device_type) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 1688, in list_logical_devices self.ensure_initialized() ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 610, in ensure_initialized pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) ^^^^^^^^^^^^^^^^^ tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server ====================================================================== ERROR: testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 (__main__.CollectiveOpsTest) CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 testAllGatherSameShape_test_axis_1_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0(axis=1, func_mode='eager', implementation=, num_processes=1, prefer_unique_instance_key=True, required_gpus=0) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 889, in testAllGatherSameShape get_global_mpr(num_processes).run(replica_fn) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1003, in run six.reraise(*process_status.exc_info) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/six_archive/six.py", line 719, in reraise raise value File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained return_value = fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 861, in replica_fn collective, devices, _ = self.make_collective(num_processes, ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 185, in make_collective collective = cross_device_ops_lib.CollectiveAllReduce( ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops.py", line 1089, in __init__ self._devices = tuple(device_util.canonicalize(d) for d in devices) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops.py", line 1089, in self._devices = tuple(device_util.canonicalize(d) for d in devices) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/device_util.py", line 60, in canonicalize config.list_logical_devices("CPU")[0].name) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/config.py", line 491, in list_logical_devices return context.context().list_logical_devices(device_type=device_type) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 1688, in list_logical_devices self.ensure_initialized() ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 610, in ensure_initialized pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) ^^^^^^^^^^^^^^^^^ tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server ====================================================================== ERROR: testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 (__main__.CollectiveOpsTest) CollectiveOpsTest.testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 testAllGatherSameShape_test_axis_1_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0(axis=1, func_mode='func_graph', implementation=, num_processes=1, prefer_unique_instance_key=True, required_gpus=0) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 889, in testAllGatherSameShape get_global_mpr(num_processes).run(replica_fn) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1003, in run six.reraise(*process_status.exc_info) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/six_archive/six.py", line 719, in reraise raise value File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained return_value = fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 861, in replica_fn collective, devices, _ = self.make_collective(num_processes, ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 185, in make_collective collective = cross_device_ops_lib.CollectiveAllReduce( ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops.py", line 1089, in __init__ self._devices = tuple(device_util.canonicalize(d) for d in devices) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops.py", line 1089, in self._devices = tuple(device_util.canonicalize(d) for d in devices) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/device_util.py", line 60, in canonicalize config.list_logical_devices("CPU")[0].name) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/config.py", line 491, in list_logical_devices return context.context().list_logical_devices(device_type=device_type) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 1688, in list_logical_devices self.ensure_initialized() ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 610, in ensure_initialized pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) ^^^^^^^^^^^^^^^^^ tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server ====================================================================== ERROR: testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 (__main__.CollectiveOpsTest) CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 testAllGatherSameShape_test_axis_2_funcmode_eager_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0(axis=2, func_mode='eager', implementation=, num_processes=1, prefer_unique_instance_key=True, required_gpus=0) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 889, in testAllGatherSameShape get_global_mpr(num_processes).run(replica_fn) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1003, in run six.reraise(*process_status.exc_info) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/six_archive/six.py", line 719, in reraise raise value File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained return_value = fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 861, in replica_fn collective, devices, _ = self.make_collective(num_processes, ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 185, in make_collective collective = cross_device_ops_lib.CollectiveAllReduce( ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops.py", line 1089, in __init__ self._devices = tuple(device_util.canonicalize(d) for d in devices) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops.py", line 1089, in self._devices = tuple(device_util.canonicalize(d) for d in devices) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/device_util.py", line 60, in canonicalize config.list_logical_devices("CPU")[0].name) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/config.py", line 491, in list_logical_devices return context.context().list_logical_devices(device_type=device_type) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 1688, in list_logical_devices self.ensure_initialized() ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 610, in ensure_initialized pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) ^^^^^^^^^^^^^^^^^ tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server ====================================================================== ERROR: testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 (__main__.CollectiveOpsTest) CollectiveOpsTest.testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0 testAllGatherSameShape_test_axis_2_funcmode_funcgraph_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_requiredgpus_0(axis=2, func_mode='func_graph', implementation=, num_processes=1, prefer_unique_instance_key=True, required_gpus=0) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 889, in testAllGatherSameShape get_global_mpr(num_processes).run(replica_fn) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1003, in run six.reraise(*process_status.exc_info) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/six_archive/six.py", line 719, in reraise raise value File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained return_value = fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 861, in replica_fn collective, devices, _ = self.make_collective(num_processes, ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 185, in make_collective collective = cross_device_ops_lib.CollectiveAllReduce( ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops.py", line 1089, in __init__ self._devices = tuple(device_util.canonicalize(d) for d in devices) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops.py", line 1089, in self._devices = tuple(device_util.canonicalize(d) for d in devices) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/device_util.py", line 60, in canonicalize config.list_logical_devices("CPU")[0].name) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/config.py", line 491, in list_logical_devices return context.context().list_logical_devices(device_type=device_type) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 1688, in list_logical_devices self.ensure_initialized() ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 610, in ensure_initialized pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) ^^^^^^^^^^^^^^^^^ tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server ====================================================================== ERROR: testAllReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_reduceop_ReduceOpSUM_requiredgpus_0 (__main__.CollectiveOpsTest) CollectiveOpsTest.testAllReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_reduceop_ReduceOpSUM_requiredgpus_0 testAllReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_reduceop_ReduceOpSUM_requiredgpus_0(implementation=, num_processes=1, reduce_op=, required_gpus=0) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 697, in testAllReduceDense get_global_mpr(num_processes).run(replica_fn) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1003, in run six.reraise(*process_status.exc_info) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/six_archive/six.py", line 719, in reraise raise value File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained return_value = fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 658, in replica_fn collective, devices, _ = self.make_collective(num_processes, ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 185, in make_collective collective = cross_device_ops_lib.CollectiveAllReduce( ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops.py", line 1089, in __init__ self._devices = tuple(device_util.canonicalize(d) for d in devices) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops.py", line 1089, in self._devices = tuple(device_util.canonicalize(d) for d in devices) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/device_util.py", line 60, in canonicalize config.list_logical_devices("CPU")[0].name) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/config.py", line 491, in list_logical_devices return context.context().list_logical_devices(device_type=device_type) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 1688, in list_logical_devices self.ensure_initialized() ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 610, in ensure_initialized pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) ^^^^^^^^^^^^^^^^^ tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server ====================================================================== ERROR: testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 (__main__.CollectiveOpsTest) CollectiveOpsTest.testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0 testBatchReduceDense_test_implementation_CommunicationImplementationRING_numprocesses_1_preferuniqueinstancekey_True_reduceop_ReduceOpMEAN_requiredgpus_0(implementation=, num_processes=1, prefer_unique_instance_key=True, reduce_op=, required_gpus=0) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 499, in testBatchReduceDense self.batch_reduce_and_verify(inputs, expect, options) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 320, in batch_reduce_and_verify get_global_mpr(options.num_processes).run(replica_fn) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1003, in run six.reraise(*process_status.exc_info) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/six_archive/six.py", line 719, in reraise raise value File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained return_value = fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 285, in replica_fn collective, devices, pid = self.make_collective(options.num_processes, ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops_test.py", line 185, in make_collective collective = cross_device_ops_lib.CollectiveAllReduce( ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops.py", line 1089, in __init__ self._devices = tuple(device_util.canonicalize(d) for d in devices) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/cross_device_ops.py", line 1089, in self._devices = tuple(device_util.canonicalize(d) for d in devices) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/distribute/device_util.py", line 60, in canonicalize config.list_logical_devices("CPU")[0].name) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/framework/config.py", line 491, in list_logical_devices return context.context().list_logical_devices(device_type=device_type) ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 1688, in list_logical_devices self.ensure_initialized() ^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/cross_device_ops_test_2gpu.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 610, in ensure_initialized pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) ^^^^^^^^^^^^^^^^^ tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server ---------------------------------------------------------------------- Ran 139 tests in 18.867s FAILED (errors=8, skipped=121) ================================================================================ ==================== Test output for //tensorflow/python/distribute:input_lib_test_cpu (shard 1 of 10): Running tests under Python 3.11.3: /usr/local/bin/python3 [ RUN ] DistributedIteratorPerDeviceTest.testDevicePlacementForInvalidCombinations_test_distribution_Mirrored2CPUs_inputoptions_InputOptionsexperimentalfetchtodeviceFalseexperimentalreplicationmodeInputReplicationModePERWORKERPERWORKERexperimentalplacedatasetondeviceTrueexperimentalperreplicabuffersize1_mode_eager INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1') I0514 22:08:37.623874 281473605792640 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1') WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/input_lib_test.py:1710: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version. Instructions for updating: rename to distribute_datasets_from_function W0514 22:08:37.689442 281473605792640 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/input_lib_test.py:1710: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version. Instructions for updating: rename to distribute_datasets_from_function INFO:tensorflow:time(__main__.DistributedIteratorPerDeviceTest.testDevicePlacementForInvalidCombinations_test_distribution_Mirrored2CPUs_inputoptions_InputOptionsexperimentalfetchtodeviceFalseexperimentalreplicationmodeInputReplicationModePERWORKERPERWORKERexperimentalplacedatasetondeviceTrueexperimentalperreplicabuffersize1_mode_eager): 0.08s I0514 22:08:37.690176 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorPerDeviceTest.testDevicePlacementForInvalidCombinations_test_distribution_Mirrored2CPUs_inputoptions_InputOptionsexperimentalfetchtodeviceFalseexperimentalreplicationmodeInputReplicationModePERWORKERPERWORKERexperimentalplacedatasetondeviceTrueexperimentalperreplicabuffersize1_mode_eager): 0.08s [ OK ] DistributedIteratorPerDeviceTest.testDevicePlacementForInvalidCombinations_test_distribution_Mirrored2CPUs_inputoptions_InputOptionsexperimentalfetchtodeviceFalseexperimentalreplicationmodeInputReplicationModePERWORKERPERWORKERexperimentalplacedatasetondeviceTrueexperimentalperreplicabuffersize1_mode_eager [ RUN ] DistributedIteratorPerDeviceTest.testDevicePlacementForPerWorkerValuesWithPrefetch_test_distribution_Mirrored2GPUsNoMergeCall_inputoptions_InputOptionsexperimentalfetchtodeviceTrueexperimentalreplicationmodeInputReplicationModePERREPLICAPERREPLICAexperimentalplacedatasetondeviceFalseexperimentalperreplicabuffersize1_mode_eager [ SKIPPED ] DistributedIteratorPerDeviceTest.testDevicePlacementForPerWorkerValuesWithPrefetch_test_distribution_Mirrored2GPUsNoMergeCall_inputoptions_InputOptionsexperimentalfetchtodeviceTrueexperimentalreplicationmodeInputReplicationModePERREPLICAPERREPLICAexperimentalplacedatasetondeviceFalseexperimentalperreplicabuffersize1_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorPerDeviceTest.testDevicePlacementForPerWorkerValuesWithPrefetch_test_distribution_Mirrored2GPUsNoMergeCall_inputoptions_InputOptionsexperimentalfetchtodeviceTrueexperimentalreplicationmodeInputReplicationModePERREPLICAPERREPLICAexperimentalplacedatasetondeviceFalseexperimentalperreplicabuffersize1_mode_eager): 0.0s I0514 22:08:37.691491 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorPerDeviceTest.testDevicePlacementForPerWorkerValuesWithPrefetch_test_distribution_Mirrored2GPUsNoMergeCall_inputoptions_InputOptionsexperimentalfetchtodeviceTrueexperimentalreplicationmodeInputReplicationModePERREPLICAPERREPLICAexperimentalplacedatasetondeviceFalseexperimentalperreplicabuffersize1_mode_eager): 0.0s [ RUN ] DistributedIteratorPerDeviceTest.testOutputValuesForPerReplicaInputOptions_test_distribution_Mirrored2CPUs_inputoptions_InputOptionsexperimentalfetchtodeviceFalseexperimentalreplicationmodeInputReplicationModePERREPLICAPERREPLICAexperimentalplacedatasetondeviceFalseexperimentalperreplicabuffersize1_mode_eager INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1') I0514 22:08:37.692564 281473605792640 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1') 2023-05-14 22:08:37.712861: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype int64 and shape [9] [[{{node Placeholder/_0}}]] 2023-05-14 22:08:37.730173: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype int64 and shape [9] [[{{node Placeholder/_0}}]] INFO:tensorflow:time(__main__.DistributedIteratorPerDeviceTest.testOutputValuesForPerReplicaInputOptions_test_distribution_Mirrored2CPUs_inputoptions_InputOptionsexperimentalfetchtodeviceFalseexperimentalreplicationmodeInputReplicationModePERREPLICAPERREPLICAexperimentalplacedatasetondeviceFalseexperimentalperreplicabuffersize1_mode_eager): 0.11s I0514 22:08:37.800121 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorPerDeviceTest.testOutputValuesForPerReplicaInputOptions_test_distribution_Mirrored2CPUs_inputoptions_InputOptionsexperimentalfetchtodeviceFalseexperimentalreplicationmodeInputReplicationModePERREPLICAPERREPLICAexperimentalplacedatasetondeviceFalseexperimentalperreplicabuffersize1_mode_eager): 0.11s [ OK ] DistributedIteratorPerDeviceTest.testOutputValuesForPerReplicaInputOptions_test_distribution_Mirrored2CPUs_inputoptions_InputOptionsexperimentalfetchtodeviceFalseexperimentalreplicationmodeInputReplicationModePERREPLICAPERREPLICAexperimentalplacedatasetondeviceFalseexperimentalperreplicabuffersize1_mode_eager [ RUN ] DistributedIteratorPerDeviceTest.testOutputValuesForPerReplicaInputOptions_test_distribution_MirroredCPUAndGPU_inputoptions_InputOptionsexperimentalfetchtodeviceFalseexperimentalreplicationmodeInputReplicationModePERREPLICAPERREPLICAexperimentalplacedatasetondeviceTrueexperimentalperreplicabuffersize1_mode_eager [ SKIPPED ] DistributedIteratorPerDeviceTest.testOutputValuesForPerReplicaInputOptions_test_distribution_MirroredCPUAndGPU_inputoptions_InputOptionsexperimentalfetchtodeviceFalseexperimentalreplicationmodeInputReplicationModePERREPLICAPERREPLICAexperimentalplacedatasetondeviceTrueexperimentalperreplicabuffersize1_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorPerDeviceTest.testOutputValuesForPerReplicaInputOptions_test_distribution_MirroredCPUAndGPU_inputoptions_InputOptionsexperimentalfetchtodeviceFalseexperimentalreplicationmodeInputReplicationModePERREPLICAPERREPLICAexperimentalplacedatasetondeviceTrueexperimentalperreplicabuffersize1_mode_eager): 0.0s I0514 22:08:37.802033 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorPerDeviceTest.testOutputValuesForPerReplicaInputOptions_test_distribution_MirroredCPUAndGPU_inputoptions_InputOptionsexperimentalfetchtodeviceFalseexperimentalreplicationmodeInputReplicationModePERREPLICAPERREPLICAexperimentalplacedatasetondeviceTrueexperimentalperreplicabuffersize1_mode_eager): 0.0s [ RUN ] DistributedIteratorPerDeviceTest.testPrefetchBufferSizeInputOptions_test_distribution_Mirrored2CPUs_inputoptions_InputOptionsexperimentalfetchtodeviceFalseexperimentalreplicationmodeInputReplicationModePERWORKERPERWORKERexperimentalplacedatasetondeviceFalseexperimentalperreplicabuffersize2_mode_eager INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1') I0514 22:08:37.803270 281473605792640 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1') 2023-05-14 22:08:37.825933: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype int64 and shape [2,5] [[{{node Placeholder/_0}}]] 2023-05-14 22:08:38.050273: W tensorflow/core/framework/dataset.cc:956] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations. 2023-05-14 22:08:38.050870: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] 2023-05-14 22:08:38.140949: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] 2023-05-14 22:08:38.333641: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Function was cancelled before it was started [[{{node RemoteCall}}]] INFO:tensorflow:time(__main__.DistributedIteratorPerDeviceTest.testPrefetchBufferSizeInputOptions_test_distribution_Mirrored2CPUs_inputoptions_InputOptionsexperimentalfetchtodeviceFalseexperimentalreplicationmodeInputReplicationModePERWORKERPERWORKERexperimentalplacedatasetondeviceFalseexperimentalperreplicabuffersize2_mode_eager): 0.72s I0514 22:08:38.518599 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorPerDeviceTest.testPrefetchBufferSizeInputOptions_test_distribution_Mirrored2CPUs_inputoptions_InputOptionsexperimentalfetchtodeviceFalseexperimentalreplicationmodeInputReplicationModePERWORKERPERWORKERexperimentalplacedatasetondeviceFalseexperimentalperreplicabuffersize2_mode_eager): 0.72s [ OK ] DistributedIteratorPerDeviceTest.testPrefetchBufferSizeInputOptions_test_distribution_Mirrored2CPUs_inputoptions_InputOptionsexperimentalfetchtodeviceFalseexperimentalreplicationmodeInputReplicationModePERWORKERPERWORKERexperimentalplacedatasetondeviceFalseexperimentalperreplicabuffersize2_mode_eager [ RUN ] DistributedIteratorTensorTypeTest.testMWMSPartialBatchWithLegacyRebatch_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_getnext_mode_eager INFO:tensorflow:Using local port 18069 I0514 22:08:38.523514 281473605792640 test_util.py:3796] Using local port 18069 INFO:tensorflow:Using local port 23609 I0514 22:08:38.524026 281473605792640 test_util.py:3796] Using local port 23609 W0514 22:08:38.524481 281473605792640 context.py:773] Configuring coordination service type may not be effective because the context is already initialized. WARNING:tensorflow:Collective ops is not configured at program startup. Some performance features may not be enabled. W0514 22:08:38.524735 281473605792640 collective_all_reduce_strategy.py:394] Collective ops is not configured at program startup. Some performance features may not be enabled. INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',) I0514 22:08:38.531927 281473605792640 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/device:CPU:0',) INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO I0514 22:08:38.532253 281473605792640 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO I0514 22:08:41.749917 281473605792640 multi_process_runner.py:989] Waiting for the result from chief-0 [chief-0]: I0514 22:08:41.764011 281473666478976 multi_process_runner.py:840] Subprocess with PID 728104 (chief, 0) is now being started. [chief-0]: I0514 22:08:41.764339 281473666478976 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"chief": ["localhost:18069"], "worker": ["localhost:23609"]}, "task": {"type": "chief", "index": 0}, "rpc_layer": "grpc"}' [worker-0]: I0514 22:08:41.780546 281473666478976 multi_process_runner.py:840] Subprocess with PID 728134 (worker, 0) is now being started. [worker-0]: I0514 22:08:41.780977 281473666478976 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"chief": ["localhost:18069"], "worker": ["localhost:23609"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [chief-0]: E0514 22:08:41.791942284 728104 server_chttp2.cc:40] {"created":"@1684102121.791810096","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1684102121.791804605","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1684102121.791777892","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684102121.791763666","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1684102121.791803585","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1684102121.791797339","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]} [chief-0]: 2023-05-14 22:08:41.792090: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:608] UNKNOWN: Could not start gRPC server [chief-0]: 2023-05-14 22:08:41.792380: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:699] Could not start gRPC server [chief-0]: Process _Process-3: [chief-0]: Traceback (most recent call last): [chief-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/strategy_combinations.py", line 208, in skip_if_cannot_start_grpc_server [chief-0]: return _create_multi_worker_mirrored() [chief-0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [chief-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/strategy_combinations.py", line 190, in _create_multi_worker_mirrored [chief-0]: strategy = CollectiveAllReduceStrategy(cluster_resolver=resolver) [chief-0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [chief-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 186, in __init__ [chief-0]: CollectiveAllReduceExtended( [chief-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 339, in __init__ [chief-0]: self._initialize_strategy(self._cluster_resolver, devices=devices) [chief-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 358, in _initialize_strategy [chief-0]: self._initialize_multi_worker(cluster_resolver) [chief-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 530, in _initialize_multi_worker [chief-0]: context.context().ensure_initialized() [chief-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 610, in ensure_initialized [chief-0]: pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) [chief-0]: tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server [chief-0]: [chief-0]: During handling of the above exception, another exception occurred: [chief-0]: [chief-0]: Traceback (most recent call last): [chief-0]: File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap [chief-0]: self.run() [chief-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 755, in _run_with_setenv [chief-0]: return self._actual_run() [chief-0]: ^^^^^^^^^^^^^^^^^^ [chief-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl [chief-0]: app.run(lambda _: self._run_impl()) [chief-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/absl_py/absl/app.py", line 312, in run [chief-0]: _run_main(main, args) [chief-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/absl_py/absl/app.py", line 258, in _run_main [chief-0]: sys.exit(main(argv)) [chief-0]: ^^^^^^^^^^ [chief-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in [chief-0]: app.run(lambda _: self._run_impl()) [chief-0]: ^^^^^^^^^^^^^^^^ [chief-0]: File "/usr/lib/python3.11/multiprocessing/process.py", line 108, in run [chief-0]: self._target(*self._args, **self._kwargs) [chief-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 866, in __call__ [chief-0]: six.reraise(*info.exc_info) [chief-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/six_archive/six.py", line 719, in reraise [chief-0]: raise value [chief-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained [chief-0]: return_value = fn(*args, **kwargs) [chief-0]: ^^^^^^^^^^^^^^^^^^^ [chief-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1026, in _pool_runner_worker [chief-0]: initializer() [chief-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/strategy_combinations.py", line 212, in skip_if_cannot_start_grpc_server [chief-0]: raise unittest.SkipTest("Cannot start std servers.") [chief-0]: unittest.case.SkipTest: Cannot start std servers. [worker-0]: 2023-05-14 22:08:41.835189: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:457] Started server with target: grpc://localhost:23609 [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1'] [worker-0]: I0514 22:08:41.841392 281473666478976 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1'] [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0514 22:08:41.926558 281473666478976 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0514 22:08:41.926924 281473666478976 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'chief': ['localhost:18069'], 'worker': ['localhost:23609']}, task_type = 'worker', task_id = 0, num_workers = 2, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0514 22:08:41.927123 281473666478976 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'chief': ['localhost:18069'], 'worker': ['localhost:23609']}, task_type = 'worker', task_id = 0, num_workers = 2, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [ FAILED ] DistributedIteratorTensorTypeTest.testMWMSPartialBatchWithLegacyRebatch_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_getnext_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTensorTypeTest.testMWMSPartialBatchWithLegacyRebatch_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_getnext_mode_eager): 3.47s I0514 22:08:41.986829 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTensorTypeTest.testMWMSPartialBatchWithLegacyRebatch_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_getnext_mode_eager): 3.47s [ RUN ] DistributedIteratorTensorTypeTest.testMWMSPartialBatch_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1GPU_inputtype_dataset_iterationtype_getnext_mode_eager [ SKIPPED ] DistributedIteratorTensorTypeTest.testMWMSPartialBatch_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1GPU_inputtype_dataset_iterationtype_getnext_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTensorTypeTest.testMWMSPartialBatch_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1GPU_inputtype_dataset_iterationtype_getnext_mode_eager): 0.0s I0514 22:08:41.988041 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTensorTypeTest.testMWMSPartialBatch_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1GPU_inputtype_dataset_iterationtype_getnext_mode_eager): 0.0s [ RUN ] DistributedIteratorTensorTypeTest.testMWMSWithDataSharding_test_apitype_wrapintodataset_autoshardpolicy_2_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_getnext_mode_eager W0514 22:08:41.988911 281473605792640 context.py:773] Configuring coordination service type may not be effective because the context is already initialized. WARNING:tensorflow:Collective ops is not configured at program startup. Some performance features may not be enabled. W0514 22:08:41.989157 281473605792640 collective_all_reduce_strategy.py:394] Collective ops is not configured at program startup. Some performance features may not be enabled. INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',) I0514 22:08:41.995753 281473605792640 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/device:CPU:0',) INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO I0514 22:08:41.996049 281473605792640 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO [ FAILED ] DistributedIteratorTensorTypeTest.testMWMSWithDataSharding_test_apitype_wrapintodataset_autoshardpolicy_2_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_getnext_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTensorTypeTest.testMWMSWithDataSharding_test_apitype_wrapintodataset_autoshardpolicy_2_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_getnext_mode_eager): 0.01s I0514 22:08:41.999925 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTensorTypeTest.testMWMSWithDataSharding_test_apitype_wrapintodataset_autoshardpolicy_2_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_getnext_mode_eager): 0.01s [ RUN ] DistributedIteratorTensorTypeTest.testMWMSWithDataSharding_test_apitype_wrapintoiterator_autoshardpolicy_2_distribution_MultiWorkerMirrored2x1GPU_inputtype_dataset_iterationtype_getnext_mode_eager [ SKIPPED ] DistributedIteratorTensorTypeTest.testMWMSWithDataSharding_test_apitype_wrapintoiterator_autoshardpolicy_2_distribution_MultiWorkerMirrored2x1GPU_inputtype_dataset_iterationtype_getnext_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTensorTypeTest.testMWMSWithDataSharding_test_apitype_wrapintoiterator_autoshardpolicy_2_distribution_MultiWorkerMirrored2x1GPU_inputtype_dataset_iterationtype_getnext_mode_eager): 0.0s I0514 22:08:42.001094 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTensorTypeTest.testMWMSWithDataSharding_test_apitype_wrapintoiterator_autoshardpolicy_2_distribution_MultiWorkerMirrored2x1GPU_inputtype_dataset_iterationtype_getnext_mode_eager): 0.0s [ RUN ] DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptionalInLoop_test_distribution_MultiWorkerMirrored2x1CPU_dropremainder_False_inputtype_inputfn_mode_eager_tfapiversion_2 W0514 22:08:42.001951 281473605792640 context.py:773] Configuring coordination service type may not be effective because the context is already initialized. WARNING:tensorflow:Collective ops is not configured at program startup. Some performance features may not be enabled. W0514 22:08:42.002177 281473605792640 collective_all_reduce_strategy.py:394] Collective ops is not configured at program startup. Some performance features may not be enabled. INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',) I0514 22:08:42.009143 281473605792640 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/device:CPU:0',) INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO I0514 22:08:42.009447 281473605792640 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO [ FAILED ] DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptionalInLoop_test_distribution_MultiWorkerMirrored2x1CPU_dropremainder_False_inputtype_inputfn_mode_eager_tfapiversion_2 INFO:tensorflow:time(__main__.DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptionalInLoop_test_distribution_MultiWorkerMirrored2x1CPU_dropremainder_False_inputtype_inputfn_mode_eager_tfapiversion_2): 0.01s I0514 22:08:42.013169 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptionalInLoop_test_distribution_MultiWorkerMirrored2x1CPU_dropremainder_False_inputtype_inputfn_mode_eager_tfapiversion_2): 0.01s [ RUN ] DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptionalInLoop_test_distribution_MultiWorkerMirrored2x2GPU_dropremainder_True_inputtype_inputfn_mode_eager_tfapiversion_2 [ SKIPPED ] DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptionalInLoop_test_distribution_MultiWorkerMirrored2x2GPU_dropremainder_True_inputtype_inputfn_mode_eager_tfapiversion_2 INFO:tensorflow:time(__main__.DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptionalInLoop_test_distribution_MultiWorkerMirrored2x2GPU_dropremainder_True_inputtype_inputfn_mode_eager_tfapiversion_2): 0.0s I0514 22:08:42.014238 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptionalInLoop_test_distribution_MultiWorkerMirrored2x2GPU_dropremainder_True_inputtype_inputfn_mode_eager_tfapiversion_2): 0.0s [ RUN ] DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_CentralStorageCPUAndGPU_dropremainder_True_enablegetnextasoptional_False_inputtype_dataset_mode_eager_tensortype_sparse [ SKIPPED ] DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_CentralStorageCPUAndGPU_dropremainder_True_enablegetnextasoptional_False_inputtype_dataset_mode_eager_tensortype_sparse INFO:tensorflow:time(__main__.DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_CentralStorageCPUAndGPU_dropremainder_True_enablegetnextasoptional_False_inputtype_dataset_mode_eager_tensortype_sparse): 0.0s I0514 22:08:42.014796 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_CentralStorageCPUAndGPU_dropremainder_True_enablegetnextasoptional_False_inputtype_dataset_mode_eager_tensortype_sparse): 0.0s [ RUN ] DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_Mirrored1CPU_dropremainder_False_enablegetnextasoptional_False_inputtype_inputfn_mode_eager_tensortype_sparse INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) I0514 22:08:42.015643 281473605792640 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) 2023-05-14 22:08:42.076698: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant and shape [20,3] [[{{node Placeholder/_0}}]] 2023-05-14 22:08:42.196813: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] INFO:tensorflow:time(__main__.DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_Mirrored1CPU_dropremainder_False_enablegetnextasoptional_False_inputtype_inputfn_mode_eager_tensortype_sparse): 0.24s I0514 22:08:42.260031 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_Mirrored1CPU_dropremainder_False_enablegetnextasoptional_False_inputtype_inputfn_mode_eager_tensortype_sparse): 0.24s [ OK ] DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_Mirrored1CPU_dropremainder_False_enablegetnextasoptional_False_inputtype_inputfn_mode_eager_tensortype_sparse [ RUN ] DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_Mirrored1CPU_dropremainder_True_enablegetnextasoptional_True_inputtype_dataset_mode_eager_tensortype_sparse INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) I0514 22:08:42.263308 281473605792640 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) 2023-05-14 22:08:42.314275: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant and shape [20,3] [[{{node Placeholder/_0}}]] 2023-05-14 22:08:42.316361: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:787] AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Found an unshardable source dataset: name: "TensorSliceDataset/_1" op: "TensorSliceDataset" input: "Placeholder/_0" attr { key: "Toutput_types" value { list { type: DT_VARIANT } } } attr { key: "_cardinality" value { i: 20 } } attr { key: "is_files" value { b: false } } attr { key: "metadata" value { s: "\n\025TensorSliceDataset:24" } } attr { key: "output_shapes" value { list { shape { dim { size: 3 } } } } } attr { key: "replicate_on_split" value { b: false } } experimental_type { type_id: TFT_PRODUCT args { type_id: TFT_DATASET args { type_id: TFT_PRODUCT args { type_id: TFT_TENSOR args { type_id: TFT_LEGACY_VARIANT } } } } } 2023-05-14 22:08:42.328217: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant and shape [20,3] [[{{node Placeholder/_0}}]] 2023-05-14 22:08:42.328779: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant and shape [20,3] [[{{node Placeholder/_0}}]] 2023-05-14 22:08:42.399674: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] INFO:tensorflow:time(__main__.DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_Mirrored1CPU_dropremainder_True_enablegetnextasoptional_True_inputtype_dataset_mode_eager_tensortype_sparse): 0.2s I0514 22:08:42.467005 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_Mirrored1CPU_dropremainder_True_enablegetnextasoptional_True_inputtype_dataset_mode_eager_tensortype_sparse): 0.2s [ OK ] DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_Mirrored1CPU_dropremainder_True_enablegetnextasoptional_True_inputtype_dataset_mode_eager_tensortype_sparse [ RUN ] DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_MirroredCPUAndGPU_dropremainder_False_enablegetnextasoptional_True_inputtype_inputfn_mode_eager_tensortype_sparse [ SKIPPED ] DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_MirroredCPUAndGPU_dropremainder_False_enablegetnextasoptional_True_inputtype_inputfn_mode_eager_tensortype_sparse INFO:tensorflow:time(__main__.DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_MirroredCPUAndGPU_dropremainder_False_enablegetnextasoptional_True_inputtype_inputfn_mode_eager_tensortype_sparse): 0.0s I0514 22:08:42.470499 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_MirroredCPUAndGPU_dropremainder_False_enablegetnextasoptional_True_inputtype_inputfn_mode_eager_tensortype_sparse): 0.0s [ RUN ] DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_OneDeviceCPU_dropremainder_False_enablegetnextasoptional_False_inputtype_dataset_mode_eager_tensortype_sparse 2023-05-14 22:08:42.531818: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant and shape [20,3] [[{{node Placeholder/_0}}]] 2023-05-14 22:08:42.532625: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:787] AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Found an unshardable source dataset: name: "TensorSliceDataset/_1" op: "TensorSliceDataset" input: "Placeholder/_0" attr { key: "Toutput_types" value { list { type: DT_VARIANT } } } attr { key: "_cardinality" value { i: 20 } } attr { key: "is_files" value { b: false } } attr { key: "metadata" value { s: "\n\025TensorSliceDataset:35" } } attr { key: "output_shapes" value { list { shape { dim { size: 3 } } } } } attr { key: "replicate_on_split" value { b: false } } experimental_type { type_id: TFT_PRODUCT args { type_id: TFT_DATASET args { type_id: TFT_PRODUCT args { type_id: TFT_TENSOR args { type_id: TFT_LEGACY_VARIANT } } } } } 2023-05-14 22:08:42.545061: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant and shape [20,3] [[{{node Placeholder/_0}}]] 2023-05-14 22:08:42.545622: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant and shape [20,3] [[{{node Placeholder/_0}}]] 2023-05-14 22:08:42.661556: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] INFO:tensorflow:time(__main__.DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_OneDeviceCPU_dropremainder_False_enablegetnextasoptional_False_inputtype_dataset_mode_eager_tensortype_sparse): 0.45s I0514 22:08:42.919579 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_OneDeviceCPU_dropremainder_False_enablegetnextasoptional_False_inputtype_dataset_mode_eager_tensortype_sparse): 0.45s [ OK ] DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_OneDeviceCPU_dropremainder_False_enablegetnextasoptional_False_inputtype_dataset_mode_eager_tensortype_sparse [ RUN ] DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_OneDeviceCPU_dropremainder_True_enablegetnextasoptional_False_inputtype_inputfn_mode_eager_tensortype_sparse 2023-05-14 22:08:42.993005: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant and shape [20,3] [[{{node Placeholder/_0}}]] 2023-05-14 22:08:43.095163: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] INFO:tensorflow:time(__main__.DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_OneDeviceCPU_dropremainder_True_enablegetnextasoptional_False_inputtype_inputfn_mode_eager_tensortype_sparse): 0.26s I0514 22:08:43.178424 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_OneDeviceCPU_dropremainder_True_enablegetnextasoptional_False_inputtype_inputfn_mode_eager_tensortype_sparse): 0.26s [ OK ] DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptional_test_distribution_OneDeviceCPU_dropremainder_True_enablegetnextasoptional_False_inputtype_inputfn_mode_eager_tensortype_sparse [ RUN ] DistributedIteratorTensorTypeTest.testRaggedSparse_test_defuntype_lambda_distribution_MirroredCPUAndGPU_dropremainder_False_inputtype_inputfn_mode_eager [ SKIPPED ] DistributedIteratorTensorTypeTest.testRaggedSparse_test_defuntype_lambda_distribution_MirroredCPUAndGPU_dropremainder_False_inputtype_inputfn_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTensorTypeTest.testRaggedSparse_test_defuntype_lambda_distribution_MirroredCPUAndGPU_dropremainder_False_inputtype_inputfn_mode_eager): 0.0s I0514 22:08:43.181388 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTensorTypeTest.testRaggedSparse_test_defuntype_lambda_distribution_MirroredCPUAndGPU_dropremainder_False_inputtype_inputfn_mode_eager): 0.0s [ RUN ] DistributedIteratorTensorTypeTest.testRaggedSparse_test_defuntype_tffunction_distribution_CentralStorageCPUAndGPU_dropremainder_True_inputtype_inputfn_mode_eager [ SKIPPED ] DistributedIteratorTensorTypeTest.testRaggedSparse_test_defuntype_tffunction_distribution_CentralStorageCPUAndGPU_dropremainder_True_inputtype_inputfn_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTensorTypeTest.testRaggedSparse_test_defuntype_tffunction_distribution_CentralStorageCPUAndGPU_dropremainder_True_inputtype_inputfn_mode_eager): 0.0s I0514 22:08:43.182216 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTensorTypeTest.testRaggedSparse_test_defuntype_tffunction_distribution_CentralStorageCPUAndGPU_dropremainder_True_inputtype_inputfn_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testAutoShardExplicit_test_distribution_Mirrored1CPU_inputtype_dataset_mode_eager INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) I0514 22:08:43.183264 281473605792640 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) 2023-05-14 22:08:43.217621: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:787] AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Found an unshardable source dataset: name: "RangeDataset/_3" op: "RangeDataset" input: "Const/_0" input: "Const/_1" input: "Const/_2" attr { key: "_cardinality" value { i: 10 } } attr { key: "metadata" value { s: "\n\017RangeDataset:55" } } attr { key: "output_shapes" value { list { shape { } } } } attr { key: "output_types" value { list { type: DT_INT64 } } } attr { key: "replicate_on_split" value { b: false } } experimental_type { type_id: TFT_PRODUCT args { type_id: TFT_DATASET args { type_id: TFT_PRODUCT args { type_id: TFT_TENSOR args { type_id: TFT_INT64 } } } } } 2023-05-14 22:08:43.223231: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:787] AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Found an unshardable source dataset: name: "RangeDataset/_3" op: "RangeDataset" input: "Const/_0" input: "Const/_1" input: "Const/_2" attr { key: "_cardinality" value { i: 10 } } attr { key: "metadata" value { s: "\n\017RangeDataset:55" } } attr { key: "output_shapes" value { list { shape { } } } } attr { key: "output_types" value { list { type: DT_INT64 } } } attr { key: "replicate_on_split" value { b: false } } experimental_type { type_id: TFT_PRODUCT args { type_id: TFT_DATASET args { type_id: TFT_PRODUCT args { type_id: TFT_TENSOR args { type_id: TFT_INT64 } } } } } 2023-05-14 22:08:43.334471: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] 2023-05-14 22:08:43.915562: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:787] AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Found an unshardable source dataset: name: "RangeDataset/_3" op: "RangeDataset" input: "Const/_0" input: "Const/_1" input: "Const/_2" attr { key: "_cardinality" value { i: 10 } } attr { key: "metadata" value { s: "\n\017RangeDataset:55" } } attr { key: "output_shapes" value { list { shape { } } } } attr { key: "output_types" value { list { type: DT_INT64 } } } attr { key: "replicate_on_split" value { b: false } } experimental_type { type_id: TFT_PRODUCT args { type_id: TFT_DATASET args { type_id: TFT_PRODUCT args { type_id: TFT_TENSOR args { type_id: TFT_INT64 } } } } } 2023-05-14 22:08:44.014379: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] INFO:tensorflow:time(__main__.DistributedIteratorTest.testAutoShardExplicit_test_distribution_Mirrored1CPU_inputtype_dataset_mode_eager): 1.15s I0514 22:08:44.328205 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testAutoShardExplicit_test_distribution_Mirrored1CPU_inputtype_dataset_mode_eager): 1.15s [ OK ] DistributedIteratorTest.testAutoShardExplicit_test_distribution_Mirrored1CPU_inputtype_dataset_mode_eager [ RUN ] DistributedIteratorTest.testAutoshardingOption_test_apitype_wrapintoiterator_autoshardpolicy_0_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_forloop_mode_eager W0514 22:08:44.331936 281473605792640 context.py:773] Configuring coordination service type may not be effective because the context is already initialized. WARNING:tensorflow:Collective ops is not configured at program startup. Some performance features may not be enabled. W0514 22:08:44.332253 281473605792640 collective_all_reduce_strategy.py:394] Collective ops is not configured at program startup. Some performance features may not be enabled. INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',) I0514 22:08:44.338236 281473605792640 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/device:CPU:0',) INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO I0514 22:08:44.338561 281473605792640 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO [ FAILED ] DistributedIteratorTest.testAutoshardingOption_test_apitype_wrapintoiterator_autoshardpolicy_0_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_forloop_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testAutoshardingOption_test_apitype_wrapintoiterator_autoshardpolicy_0_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_forloop_mode_eager): 0.01s I0514 22:08:44.342363 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testAutoshardingOption_test_apitype_wrapintoiterator_autoshardpolicy_0_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_forloop_mode_eager): 0.01s [ RUN ] DistributedIteratorTest.testBatchSplittingMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x2GPUNoMergeCall_enablegetnextasoptional_True_inputtype_dataset_iterationtype_getnext_mode_eager_numreplicasinsync_2 [ SKIPPED ] DistributedIteratorTest.testBatchSplittingMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x2GPUNoMergeCall_enablegetnextasoptional_True_inputtype_dataset_iterationtype_getnext_mode_eager_numreplicasinsync_2 INFO:tensorflow:time(__main__.DistributedIteratorTest.testBatchSplittingMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x2GPUNoMergeCall_enablegetnextasoptional_True_inputtype_dataset_iterationtype_getnext_mode_eager_numreplicasinsync_2): 0.0s I0514 22:08:44.343497 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testBatchSplittingMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x2GPUNoMergeCall_enablegetnextasoptional_True_inputtype_dataset_iterationtype_getnext_mode_eager_numreplicasinsync_2): 0.0s [ RUN ] DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager_numreplicasinsync_2 [ SKIPPED ] DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager_numreplicasinsync_2 INFO:tensorflow:time(__main__.DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager_numreplicasinsync_2): 0.0s I0514 22:08:44.344170 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager_numreplicasinsync_2): 0.0s [ RUN ] DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_forloop_mode_graph_numreplicasinsync_2 [ SKIPPED ] DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_forloop_mode_graph_numreplicasinsync_2 INFO:tensorflow:time(__main__.DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_forloop_mode_graph_numreplicasinsync_2): 0.0s I0514 22:08:44.344750 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_forloop_mode_graph_numreplicasinsync_2): 0.0s [ RUN ] DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_getnext_mode_eager_numreplicasinsync_2 [ SKIPPED ] DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_getnext_mode_eager_numreplicasinsync_2 INFO:tensorflow:time(__main__.DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_getnext_mode_eager_numreplicasinsync_2): 0.0s I0514 22:08:44.345274 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_getnext_mode_eager_numreplicasinsync_2): 0.0s [ RUN ] DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_getnext_mode_graph_numreplicasinsync_2 [ SKIPPED ] DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_getnext_mode_graph_numreplicasinsync_2 INFO:tensorflow:time(__main__.DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_getnext_mode_graph_numreplicasinsync_2): 0.0s I0514 22:08:44.345784 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_getnext_mode_graph_numreplicasinsync_2): 0.0s [ RUN ] DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_forloop_mode_eager_numreplicasinsync_2 [ SKIPPED ] DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_forloop_mode_eager_numreplicasinsync_2 INFO:tensorflow:time(__main__.DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_forloop_mode_eager_numreplicasinsync_2): 0.01s I0514 22:08:44.355590 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_forloop_mode_eager_numreplicasinsync_2): 0.01s [ RUN ] DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_graph_numreplicasinsync_2 [ SKIPPED ] DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_graph_numreplicasinsync_2 INFO:tensorflow:time(__main__.DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_graph_numreplicasinsync_2): 0.0s I0514 22:08:44.356500 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_graph_numreplicasinsync_2): 0.0s [ RUN ] DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_getnext_mode_eager_numreplicasinsync_2 [ SKIPPED ] DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_getnext_mode_eager_numreplicasinsync_2 INFO:tensorflow:time(__main__.DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_getnext_mode_eager_numreplicasinsync_2): 0.0s I0514 22:08:44.357096 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testBatchSplitting_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_getnext_mode_eager_numreplicasinsync_2): 0.0s [ RUN ] DistributedIteratorTest.testCacheAcrossIteration_test_distribution_OneDeviceCPU_mode_eager 2023-05-14 22:08:44.392952: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:787] AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Found an unshardable source dataset: name: "RangeDataset/_3" op: "RangeDataset" input: "Const/_0" input: "Const/_1" input: "Const/_2" attr { key: "_cardinality" value { i: 16 } } attr { key: "metadata" value { s: "\n\017RangeDataset:75" } } attr { key: "output_shapes" value { list { shape { } } } } attr { key: "output_types" value { list { type: DT_INT64 } } } attr { key: "replicate_on_split" value { b: false } } experimental_type { type_id: TFT_PRODUCT args { type_id: TFT_DATASET args { type_id: TFT_PRODUCT args { type_id: TFT_TENSOR args { type_id: TFT_INT64 } } } } } 2023-05-14 22:08:44.592417: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] 2023-05-14 22:08:44.790625: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] INFO:tensorflow:time(__main__.DistributedIteratorTest.testCacheAcrossIteration_test_distribution_OneDeviceCPU_mode_eager): 0.75s I0514 22:08:45.110148 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testCacheAcrossIteration_test_distribution_OneDeviceCPU_mode_eager): 0.75s [ OK ] DistributedIteratorTest.testCacheAcrossIteration_test_distribution_OneDeviceCPU_mode_eager [ RUN ] DistributedIteratorTest.testGetNextOptionalShapeEmpty_test_distribution_OneDeviceCPU_mode_eager 2023-05-14 22:08:45.157826: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_1' with dtype float and shape [8] [[{{node Placeholder/_1}}]] 2023-05-14 22:08:45.158617: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:787] AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Found an unshardable source dataset: name: "TensorSliceDataset/_2" op: "TensorSliceDataset" input: "Placeholder/_0" input: "Placeholder/_1" attr { key: "Toutput_types" value { list { type: DT_FLOAT type: DT_FLOAT } } } attr { key: "_cardinality" value { i: 8 } } attr { key: "is_files" value { b: false } } attr { key: "metadata" value { s: "\n\025TensorSliceDataset:93" } } attr { key: "output_shapes" value { list { shape { dim { size: 10 } } shape { } } } } attr { key: "replicate_on_split" value { b: false } } experimental_type { type_id: TFT_PRODUCT args { type_id: TFT_DATASET args { type_id: TFT_PRODUCT args { type_id: TFT_TENSOR args { type_id: TFT_FLOAT } } args { type_id: TFT_TENSOR args { type_id: TFT_FLOAT } } } } } /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/data/ops/dataset_ops.py:468: UserWarning: To make it possible to preserve tf.data options across serialization boundaries, their implementation has moved to be part of the TensorFlow graph. As a consequence, the options value is in general no longer known at graph construction time. Invoking this method in graph mode retains the legacy behavior of the original implementation, but note that the returned value might not reflect the actual value of the options. warnings.warn("To make it possible to preserve tf.data options across " 2023-05-14 22:08:45.599380: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_1' with dtype float and shape [8] [[{{node Placeholder/_1}}]] 2023-05-14 22:08:45.635248: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] I0514 22:08:45.746275 281451815170528 multi_process_runner.py:715] chief-0 has already exited. Not terminating. I0514 22:08:45.747373 281451815170528 multi_process_runner.py:721] worker-0 terminated with signal . 2023-05-14 22:08:45.860057: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Function was cancelled before it was started [[{{node RemoteCall}}]] INFO:tensorflow:time(__main__.DistributedIteratorTest.testGetNextOptionalShapeEmpty_test_distribution_OneDeviceCPU_mode_eager): 0.77s I0514 22:08:45.881484 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testGetNextOptionalShapeEmpty_test_distribution_OneDeviceCPU_mode_eager): 0.77s [ OK ] DistributedIteratorTest.testGetNextOptionalShapeEmpty_test_distribution_OneDeviceCPU_mode_eager [ RUN ] DistributedIteratorTest.testGetNextOptionalShapeInfinite_test_distribution_CentralStorage2GPUs_mode_eager [ SKIPPED ] DistributedIteratorTest.testGetNextOptionalShapeInfinite_test_distribution_CentralStorage2GPUs_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testGetNextOptionalShapeInfinite_test_distribution_CentralStorage2GPUs_mode_eager): 0.0s I0514 22:08:45.883510 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testGetNextOptionalShapeInfinite_test_distribution_CentralStorage2GPUs_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testIterableIterator_test_distribution_Mirrored1CPU_mode_eager INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) I0514 22:08:45.884524 281473605792640 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) 2023-05-14 22:08:45.923933: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:787] AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Found an unshardable source dataset: name: "RangeDataset/_3" op: "RangeDataset" input: "Const/_0" input: "Const/_1" input: "Const/_2" attr { key: "_cardinality" value { i: 10 } } attr { key: "metadata" value { s: "\n\020RangeDataset:104" } } attr { key: "output_shapes" value { list { shape { } } } } attr { key: "output_types" value { list { type: DT_INT64 } } } attr { key: "replicate_on_split" value { b: false } } experimental_type { type_id: TFT_PRODUCT args { type_id: TFT_DATASET args { type_id: TFT_PRODUCT args { type_id: TFT_TENSOR args { type_id: TFT_INT64 } } } } } 2023-05-14 22:08:46.008241: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] INFO:tensorflow:time(__main__.DistributedIteratorTest.testIterableIterator_test_distribution_Mirrored1CPU_mode_eager): 0.39s I0514 22:08:46.278247 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testIterableIterator_test_distribution_Mirrored1CPU_mode_eager): 0.39s [ OK ] DistributedIteratorTest.testIterableIterator_test_distribution_Mirrored1CPU_mode_eager [ RUN ] DistributedIteratorTest.testOneDeviceCPUMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager W0514 22:08:46.280904 281473605792640 context.py:773] Configuring coordination service type may not be effective because the context is already initialized. WARNING:tensorflow:Collective ops is not configured at program startup. Some performance features may not be enabled. W0514 22:08:46.281200 281473605792640 collective_all_reduce_strategy.py:394] Collective ops is not configured at program startup. Some performance features may not be enabled. INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',) I0514 22:08:46.286836 281473605792640 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/device:CPU:0',) INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO I0514 22:08:46.287139 281473605792640 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO [ FAILED ] DistributedIteratorTest.testOneDeviceCPUMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testOneDeviceCPUMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager): 0.01s I0514 22:08:46.290779 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testOneDeviceCPUMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager): 0.01s [ RUN ] DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintodataset_distribution_Mirrored1CPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_getnext_mode_eager INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) I0514 22:08:46.292078 281473605792640 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) 2023-05-14 22:08:46.312395: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:787] AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Found an unshardable source dataset: name: "RangeDataset/_3" op: "RangeDataset" input: "Const/_0" input: "Const/_1" input: "Const/_2" attr { key: "_cardinality" value { i: 10 } } attr { key: "metadata" value { s: "\n\020RangeDataset:113" } } attr { key: "output_shapes" value { list { shape { } } } } attr { key: "output_types" value { list { type: DT_INT64 } } } attr { key: "replicate_on_split" value { b: false } } experimental_type { type_id: TFT_PRODUCT args { type_id: TFT_DATASET args { type_id: TFT_PRODUCT args { type_id: TFT_TENSOR args { type_id: TFT_INT64 } } } } } 2023-05-14 22:08:46.398002: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] 2023-05-14 22:08:46.749469: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] 2023-05-14 22:08:47.017848: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] INFO:tensorflow:time(__main__.DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintodataset_distribution_Mirrored1CPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_getnext_mode_eager): 1.28s I0514 22:08:47.571094 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintodataset_distribution_Mirrored1CPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_getnext_mode_eager): 1.28s [ OK ] DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintodataset_distribution_Mirrored1CPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_getnext_mode_eager [ RUN ] DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintodataset_distribution_Mirrored1CPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_forloop_mode_eager INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) I0514 22:08:47.575262 281473605792640 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) 2023-05-14 22:08:47.669947: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] 2023-05-14 22:08:47.814945: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] INFO:tensorflow:time(__main__.DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintodataset_distribution_Mirrored1CPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.41s I0514 22:08:47.988981 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintodataset_distribution_Mirrored1CPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.41s [ OK ] DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintodataset_distribution_Mirrored1CPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_forloop_mode_eager [ RUN ] DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintodataset_distribution_OneDeviceCPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_getnext_mode_eager 2023-05-14 22:08:48.162417: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] 2023-05-14 22:08:48.336695: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] 2023-05-14 22:08:48.569163: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] INFO:tensorflow:time(__main__.DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintodataset_distribution_OneDeviceCPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_getnext_mode_eager): 0.88s I0514 22:08:48.868173 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintodataset_distribution_OneDeviceCPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_getnext_mode_eager): 0.88s [ OK ] DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintodataset_distribution_OneDeviceCPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_getnext_mode_eager [ RUN ] DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintoiterator_distribution_Mirrored1CPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) I0514 22:08:48.871962 281473605792640 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) [ SKIPPED ] DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintoiterator_distribution_Mirrored1CPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintoiterator_distribution_Mirrored1CPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager): 0.01s I0514 22:08:48.879727 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintoiterator_distribution_Mirrored1CPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager): 0.01s [ RUN ] DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintoiterator_distribution_Mirrored1CPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_getnext_mode_eager INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) I0514 22:08:48.881085 281473605792640 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) 2023-05-14 22:08:48.899744: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:787] AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Found an unshardable source dataset: name: "RangeDataset/_3" op: "RangeDataset" input: "Const/_0" input: "Const/_1" input: "Const/_2" attr { key: "_cardinality" value { i: 10 } } attr { key: "metadata" value { s: "\n\020RangeDataset:167" } } attr { key: "output_shapes" value { list { shape { } } } } attr { key: "output_types" value { list { type: DT_INT64 } } } attr { key: "replicate_on_split" value { b: false } } experimental_type { type_id: TFT_PRODUCT args { type_id: TFT_DATASET args { type_id: TFT_PRODUCT args { type_id: TFT_TENSOR args { type_id: TFT_INT64 } } } } } 2023-05-14 22:08:48.985124: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] [ SKIPPED ] DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintoiterator_distribution_Mirrored1CPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_getnext_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintoiterator_distribution_Mirrored1CPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_getnext_mode_eager): 0.18s I0514 22:08:49.057524 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintoiterator_distribution_Mirrored1CPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_getnext_mode_eager): 0.18s [ RUN ] DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintoiterator_distribution_OneDeviceCPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_forloop_mode_eager [ SKIPPED ] DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintoiterator_distribution_OneDeviceCPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_forloop_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintoiterator_distribution_OneDeviceCPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s I0514 22:08:49.060256 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintoiterator_distribution_OneDeviceCPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintoiterator_distribution_OneDeviceCPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_getnext_mode_eager [ SKIPPED ] DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintoiterator_distribution_OneDeviceCPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_getnext_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintoiterator_distribution_OneDeviceCPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_getnext_mode_eager): 0.0s I0514 22:08:49.061235 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testOneDeviceCPU_test_apitype_wrapintoiterator_distribution_OneDeviceCPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_getnext_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testShuffleAcrossIterations_test_distribution_MultiWorkerMirrored2x2GPUNoMergeCall_mode_eager_reshuffle_False [ SKIPPED ] DistributedIteratorTest.testShuffleAcrossIterations_test_distribution_MultiWorkerMirrored2x2GPUNoMergeCall_mode_eager_reshuffle_False INFO:tensorflow:time(__main__.DistributedIteratorTest.testShuffleAcrossIterations_test_distribution_MultiWorkerMirrored2x2GPUNoMergeCall_mode_eager_reshuffle_False): 0.0s I0514 22:08:49.061919 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testShuffleAcrossIterations_test_distribution_MultiWorkerMirrored2x2GPUNoMergeCall_mode_eager_reshuffle_False): 0.0s [ RUN ] DistributedIteratorTest.testTPU_test_apitype_wrapintodataset_distribution_TPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_getnext_mode_eager [ SKIPPED ] DistributedIteratorTest.testTPU_test_apitype_wrapintodataset_distribution_TPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_getnext_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testTPU_test_apitype_wrapintodataset_distribution_TPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_getnext_mode_eager): 0.0s I0514 22:08:49.062434 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testTPU_test_apitype_wrapintodataset_distribution_TPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_getnext_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testTPU_test_apitype_wrapintodataset_distribution_TPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_forloop_mode_eager [ SKIPPED ] DistributedIteratorTest.testTPU_test_apitype_wrapintodataset_distribution_TPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_forloop_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testTPU_test_apitype_wrapintodataset_distribution_TPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s I0514 22:08:49.062940 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testTPU_test_apitype_wrapintodataset_distribution_TPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testTPU_test_apitype_wrapintoiterator_distribution_TPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_getnext_mode_eager [ SKIPPED ] DistributedIteratorTest.testTPU_test_apitype_wrapintoiterator_distribution_TPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_getnext_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testTPU_test_apitype_wrapintoiterator_distribution_TPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_getnext_mode_eager): 0.0s I0514 22:08:49.063442 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testTPU_test_apitype_wrapintoiterator_distribution_TPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_getnext_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testTupleDatasetMultiworker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x2GPUNoMergeCall_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager [ SKIPPED ] DistributedIteratorTest.testTupleDatasetMultiworker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x2GPUNoMergeCall_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testTupleDatasetMultiworker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x2GPUNoMergeCall_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager): 0.0s I0514 22:08:49.063941 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testTupleDatasetMultiworker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x2GPUNoMergeCall_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testTupleDatasetMultiworker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x2GPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_forloop_mode_eager [ SKIPPED ] DistributedIteratorTest.testTupleDatasetMultiworker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x2GPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_forloop_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testTupleDatasetMultiworker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x2GPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s I0514 22:08:49.064435 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testTupleDatasetMultiworker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x2GPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testTupleDataset_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_forloop_mode_eager [ SKIPPED ] DistributedIteratorTest.testTupleDataset_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_forloop_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testTupleDataset_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s I0514 22:08:49.064930 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testTupleDataset_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testTupleDataset_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_getnext_mode_eager [ SKIPPED ] DistributedIteratorTest.testTupleDataset_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_getnext_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testTupleDataset_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_getnext_mode_eager): 0.0s I0514 22:08:49.065415 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testTupleDataset_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_getnext_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testTupleDataset_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_forloop_mode_eager [ SKIPPED ] DistributedIteratorTest.testTupleDataset_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_forloop_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testTupleDataset_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_forloop_mode_eager): 0.0s I0514 22:08:49.065898 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testTupleDataset_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_forloop_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testTupleDataset_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_getnext_mode_eager [ SKIPPED ] DistributedIteratorTest.testTupleDataset_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_getnext_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testTupleDataset_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_getnext_mode_eager): 0.0s I0514 22:08:49.066413 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testTupleDataset_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_getnext_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testTupleDataset_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_forloop_mode_eager [ SKIPPED ] DistributedIteratorTest.testTupleDataset_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_forloop_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testTupleDataset_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s I0514 22:08:49.066906 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testTupleDataset_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testTupleDataset_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_getnext_mode_eager [ SKIPPED ] DistributedIteratorTest.testTupleDataset_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_getnext_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testTupleDataset_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_getnext_mode_eager): 0.0s I0514 22:08:49.067395 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testTupleDataset_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_getnext_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager [ SKIPPED ] DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager): 0.0s I0514 22:08:49.067879 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_getnext_mode_eager [ SKIPPED ] DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_getnext_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_getnext_mode_eager): 0.0s I0514 22:08:49.068360 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_getnext_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_forloop_mode_eager [ SKIPPED ] DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_forloop_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s I0514 22:08:49.068849 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_enablegetnextasoptional_False_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_getnext_mode_eager [ SKIPPED ] DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_getnext_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_getnext_mode_eager): 0.0s I0514 22:08:49.069334 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_getnext_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_forloop_mode_eager [ SKIPPED ] DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_forloop_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_forloop_mode_eager): 0.0s I0514 22:08:49.069818 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_enablegetnextasoptional_True_inputtype_dataset_iterationtype_forloop_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_getnext_mode_eager [ SKIPPED ] DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_getnext_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_getnext_mode_eager): 0.0s I0514 22:08:49.070296 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_getnext_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_forloop_mode_eager [ SKIPPED ] DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_forloop_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s I0514 22:08:49.070777 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testTwoDevicesOneGPUOneCPU_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_enablegetnextasoptional_True_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testUnevenDatasetBatchesMultiWorkerFourReplicas_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x2GPUNoMergeCall_dropremainder_True_inputtype_inputfn_iterationtype_forloop_mode_eager [ SKIPPED ] DistributedIteratorTest.testUnevenDatasetBatchesMultiWorkerFourReplicas_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x2GPUNoMergeCall_dropremainder_True_inputtype_inputfn_iterationtype_forloop_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testUnevenDatasetBatchesMultiWorkerFourReplicas_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x2GPUNoMergeCall_dropremainder_True_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s I0514 22:08:49.071263 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testUnevenDatasetBatchesMultiWorkerFourReplicas_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x2GPUNoMergeCall_dropremainder_True_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testUnevenDatasetBatchesMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_dropremainder_False_inputtype_dataset_iterationtype_forloop_mode_eager W0514 22:08:49.071959 281473605792640 context.py:773] Configuring coordination service type may not be effective because the context is already initialized. WARNING:tensorflow:Collective ops is not configured at program startup. Some performance features may not be enabled. W0514 22:08:49.072178 281473605792640 collective_all_reduce_strategy.py:394] Collective ops is not configured at program startup. Some performance features may not be enabled. INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',) I0514 22:08:49.077848 281473605792640 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/device:CPU:0',) INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO I0514 22:08:49.078135 281473605792640 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO [ FAILED ] DistributedIteratorTest.testUnevenDatasetBatchesMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_dropremainder_False_inputtype_dataset_iterationtype_forloop_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testUnevenDatasetBatchesMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_dropremainder_False_inputtype_dataset_iterationtype_forloop_mode_eager): 0.01s I0514 22:08:49.081745 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testUnevenDatasetBatchesMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_dropremainder_False_inputtype_dataset_iterationtype_forloop_mode_eager): 0.01s [ RUN ] DistributedIteratorTest.testUnevenDatasetBatchesMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1GPU_dropremainder_False_inputtype_inputfn_iterationtype_forloop_mode_eager [ SKIPPED ] DistributedIteratorTest.testUnevenDatasetBatchesMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1GPU_dropremainder_False_inputtype_inputfn_iterationtype_forloop_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testUnevenDatasetBatchesMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1GPU_dropremainder_False_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s I0514 22:08:49.082749 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testUnevenDatasetBatchesMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1GPU_dropremainder_False_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_dropremainder_False_inputtype_inputfn_iterationtype_forloop_mode_eager [ SKIPPED ] DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_dropremainder_False_inputtype_inputfn_iterationtype_forloop_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_dropremainder_False_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s I0514 22:08:49.083306 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_dropremainder_False_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_dropremainder_True_inputtype_inputfn_iterationtype_getnext_mode_eager [ SKIPPED ] DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_dropremainder_True_inputtype_inputfn_iterationtype_getnext_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_dropremainder_True_inputtype_inputfn_iterationtype_getnext_mode_eager): 0.0s I0514 22:08:49.083822 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintodataset_distribution_CentralStorageCPUAndGPU_dropremainder_True_inputtype_inputfn_iterationtype_getnext_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_dropremainder_True_inputtype_dataset_iterationtype_forloop_mode_eager [ SKIPPED ] DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_dropremainder_True_inputtype_dataset_iterationtype_forloop_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_dropremainder_True_inputtype_dataset_iterationtype_forloop_mode_eager): 0.0s I0514 22:08:49.084321 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintodataset_distribution_MirroredCPUAndGPU_dropremainder_True_inputtype_dataset_iterationtype_forloop_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_dropremainder_False_inputtype_dataset_iterationtype_getnext_mode_eager [ SKIPPED ] DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_dropremainder_False_inputtype_dataset_iterationtype_getnext_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_dropremainder_False_inputtype_dataset_iterationtype_getnext_mode_eager): 0.0s I0514 22:08:49.084816 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_dropremainder_False_inputtype_dataset_iterationtype_getnext_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_dropremainder_True_inputtype_inputfn_iterationtype_forloop_mode_eager [ SKIPPED ] DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_dropremainder_True_inputtype_inputfn_iterationtype_forloop_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_dropremainder_True_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s I0514 22:08:49.085303 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintoiterator_distribution_CentralStorageCPUAndGPU_dropremainder_True_inputtype_inputfn_iterationtype_forloop_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_dropremainder_False_inputtype_inputfn_iterationtype_getnext_mode_eager [ SKIPPED ] DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_dropremainder_False_inputtype_inputfn_iterationtype_getnext_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_dropremainder_False_inputtype_inputfn_iterationtype_getnext_mode_eager): 0.0s I0514 22:08:49.085794 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTest.testUnevenDatasetBatches_test_apitype_wrapintoiterator_distribution_MirroredCPUAndGPU_dropremainder_False_inputtype_inputfn_iterationtype_getnext_mode_eager): 0.0s [ RUN ] DistributedIteratorTest.test_session [ SKIPPED ] DistributedIteratorTest.test_session [ RUN ] DistributedIteratorTfDataServiceTest.testDistributeDatasetFromFunctionNested_test_distribution_TPU_mode_eager 2023-05-14 22:08:49.087947: I tensorflow/core/data/service/dispatcher_impl.cc:204] Running with fault_tolerant_mode=False. The dispatcher will not be able to recover its state on restart. 2023-05-14 22:08:49.087994: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data DispatchServer running at 0.0.0.0:40477 2023-05-14 22:08:49.090488: I tensorflow/core/data/service/worker_impl.cc:186] Worker registered with dispatcher running at localhost:40477 2023-05-14 22:08:49.090696: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:38321 2023-05-14 22:08:49.092543: I tensorflow/core/data/service/worker_impl.cc:186] Worker registered with dispatcher running at localhost:40477 2023-05-14 22:08:49.092890: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:39287 2023-05-14 22:08:49.094650: I tensorflow/core/data/service/worker_impl.cc:186] Worker registered with dispatcher running at localhost:40477 2023-05-14 22:08:49.094839: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:35821 [ SKIPPED ] DistributedIteratorTfDataServiceTest.testDistributeDatasetFromFunctionNested_test_distribution_TPU_mode_eager INFO:tensorflow:time(__main__.DistributedIteratorTfDataServiceTest.testDistributeDatasetFromFunctionNested_test_distribution_TPU_mode_eager): 0.01s I0514 22:08:49.095342 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTfDataServiceTest.testDistributeDatasetFromFunctionNested_test_distribution_TPU_mode_eager): 0.01s [ RUN ] DistributedIteratorTfDataServiceTest.testTfDataService_test_distribution_Mirrored1CPU_mode_eager 2023-05-14 22:08:49.096468: I tensorflow/core/data/service/dispatcher_impl.cc:204] Running with fault_tolerant_mode=False. The dispatcher will not be able to recover its state on restart. 2023-05-14 22:08:49.096513: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data DispatchServer running at 0.0.0.0:35283 2023-05-14 22:08:49.099464: I tensorflow/core/data/service/worker_impl.cc:186] Worker registered with dispatcher running at localhost:35283 2023-05-14 22:08:49.099672: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:37201 2023-05-14 22:08:49.101534: I tensorflow/core/data/service/worker_impl.cc:186] Worker registered with dispatcher running at localhost:35283 2023-05-14 22:08:49.101725: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:34347 2023-05-14 22:08:49.103937: I tensorflow/core/data/service/worker_impl.cc:186] Worker registered with dispatcher running at localhost:35283 2023-05-14 22:08:49.104122: I tensorflow/core/data/service/server_lib.cc:82] Started tf.data WorkerServer running at 0.0.0.0:45989 INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) I0514 22:08:49.104862 281473605792640 mirrored_strategy.py:423] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) 2023-05-14 22:08:49.262686: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype variant [[{{node Placeholder/_0}}]] INFO:tensorflow:time(__main__.DistributedIteratorTfDataServiceTest.testTfDataService_test_distribution_Mirrored1CPU_mode_eager): 2.77s I0514 22:08:51.867401 281473605792640 test_util.py:2464] time(__main__.DistributedIteratorTfDataServiceTest.testTfDataService_test_distribution_Mirrored1CPU_mode_eager): 2.77s [ OK ] DistributedIteratorTfDataServiceTest.testTfDataService_test_distribution_Mirrored1CPU_mode_eager I0514 22:08:51.868303 281473605792640 multi_process_runner.py:646] chief-0 exit code: 1 I0514 22:08:51.868443 281473605792640 multi_process_runner.py:646] worker-0 exit code: -9 E0514 22:08:51.877152 281473605792640 multi_process_runner.py:933] Ignoring exception when shutting down MultiProcessPoolRunner: Cannot start std servers. ====================================================================== ERROR: testMWMSPartialBatchWithLegacyRebatch_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_getnext_mode_eager (__main__.DistributedIteratorTensorTypeTest) DistributedIteratorTensorTypeTest.testMWMSPartialBatchWithLegacyRebatch_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_getnext_mode_eager testMWMSPartialBatchWithLegacyRebatch_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_getnext_mode_eager(api_type='wrap_into_dataset', distribution=MultiWorkerMirrored2x1CPU, input_type='dataset', iteration_type='get_next', mode='eager') ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 580, in decorator results = runner.run(_test_runner, args=(test_id, _env)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 991, in run process_statuses.append(conn.recv()) ^^^^^^^^^^^ File "/usr/lib/python3.11/multiprocessing/connection.py", line 249, in recv buf = self._recv_bytes() ^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/multiprocessing/connection.py", line 413, in _recv_bytes buf = self._recv(4) ^^^^^^^^^^^^^ File "/usr/lib/python3.11/multiprocessing/connection.py", line 378, in _recv chunk = read(handle, remaining) ^^^^^^^^^^^^^^^^^^^^^^^ ConnectionResetError: [Errno 104] Connection reset by peer ====================================================================== ERROR: testMWMSWithDataSharding_test_apitype_wrapintodataset_autoshardpolicy_2_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_getnext_mode_eager (__main__.DistributedIteratorTensorTypeTest) DistributedIteratorTensorTypeTest.testMWMSWithDataSharding_test_apitype_wrapintodataset_autoshardpolicy_2_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_getnext_mode_eager testMWMSWithDataSharding_test_apitype_wrapintodataset_autoshardpolicy_2_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_getnext_mode_eager(api_type='wrap_into_dataset', auto_shard_policy=, distribution=MultiWorkerMirrored2x1CPU, input_type='dataset', iteration_type='get_next', mode='eager') ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 580, in decorator results = runner.run(_test_runner, args=(test_id, _env)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 985, in run conn.send((fn, args or [], kwargs or {})) File "/usr/lib/python3.11/multiprocessing/connection.py", line 205, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/usr/lib/python3.11/multiprocessing/connection.py", line 410, in _send_bytes self._send(header + buf) File "/usr/lib/python3.11/multiprocessing/connection.py", line 367, in _send n = write(self._handle, buf) ^^^^^^^^^^^^^^^^^^^^^^^^ BrokenPipeError: [Errno 32] Broken pipe ====================================================================== ERROR: testRaggedSparseGetNextAsOptionalInLoop_test_distribution_MultiWorkerMirrored2x1CPU_dropremainder_False_inputtype_inputfn_mode_eager_tfapiversion_2 (__main__.DistributedIteratorTensorTypeTest) DistributedIteratorTensorTypeTest.testRaggedSparseGetNextAsOptionalInLoop_test_distribution_MultiWorkerMirrored2x1CPU_dropremainder_False_inputtype_inputfn_mode_eager_tfapiversion_2 testRaggedSparseGetNextAsOptionalInLoop_test_distribution_MultiWorkerMirrored2x1CPU_dropremainder_False_inputtype_inputfn_mode_eager_tfapiversion_2(distribution=MultiWorkerMirrored2x1CPU, drop_remainder=False, input_type='input_fn', mode='eager', tf_api_version=2) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 580, in decorator results = runner.run(_test_runner, args=(test_id, _env)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 985, in run conn.send((fn, args or [], kwargs or {})) File "/usr/lib/python3.11/multiprocessing/connection.py", line 205, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/usr/lib/python3.11/multiprocessing/connection.py", line 410, in _send_bytes self._send(header + buf) File "/usr/lib/python3.11/multiprocessing/connection.py", line 367, in _send n = write(self._handle, buf) ^^^^^^^^^^^^^^^^^^^^^^^^ BrokenPipeError: [Errno 32] Broken pipe ====================================================================== ERROR: testAutoshardingOption_test_apitype_wrapintoiterator_autoshardpolicy_0_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_forloop_mode_eager (__main__.DistributedIteratorTest) DistributedIteratorTest.testAutoshardingOption_test_apitype_wrapintoiterator_autoshardpolicy_0_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_forloop_mode_eager testAutoshardingOption_test_apitype_wrapintoiterator_autoshardpolicy_0_distribution_MultiWorkerMirrored2x1CPU_inputtype_dataset_iterationtype_forloop_mode_eager(api_type='wrap_into_iterator', auto_shard_policy=, distribution=MultiWorkerMirrored2x1CPU, input_type='dataset', iteration_type='for_loop', mode='eager') ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 580, in decorator results = runner.run(_test_runner, args=(test_id, _env)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 985, in run conn.send((fn, args or [], kwargs or {})) File "/usr/lib/python3.11/multiprocessing/connection.py", line 205, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/usr/lib/python3.11/multiprocessing/connection.py", line 410, in _send_bytes self._send(header + buf) File "/usr/lib/python3.11/multiprocessing/connection.py", line 367, in _send n = write(self._handle, buf) ^^^^^^^^^^^^^^^^^^^^^^^^ BrokenPipeError: [Errno 32] Broken pipe ====================================================================== ERROR: testOneDeviceCPUMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager (__main__.DistributedIteratorTest) DistributedIteratorTest.testOneDeviceCPUMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager testOneDeviceCPUMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_enablegetnextasoptional_False_inputtype_dataset_iterationtype_forloop_mode_eager(api_type='wrap_into_dataset', distribution=MultiWorkerMirrored2x1CPU, enable_get_next_as_optional=False, input_type='dataset', iteration_type='for_loop', mode='eager') ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 580, in decorator results = runner.run(_test_runner, args=(test_id, _env)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 985, in run conn.send((fn, args or [], kwargs or {})) File "/usr/lib/python3.11/multiprocessing/connection.py", line 205, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/usr/lib/python3.11/multiprocessing/connection.py", line 410, in _send_bytes self._send(header + buf) File "/usr/lib/python3.11/multiprocessing/connection.py", line 367, in _send n = write(self._handle, buf) ^^^^^^^^^^^^^^^^^^^^^^^^ BrokenPipeError: [Errno 32] Broken pipe ====================================================================== ERROR: testUnevenDatasetBatchesMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_dropremainder_False_inputtype_dataset_iterationtype_forloop_mode_eager (__main__.DistributedIteratorTest) DistributedIteratorTest.testUnevenDatasetBatchesMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_dropremainder_False_inputtype_dataset_iterationtype_forloop_mode_eager testUnevenDatasetBatchesMultiWorker_test_apitype_wrapintodataset_distribution_MultiWorkerMirrored2x1CPU_dropremainder_False_inputtype_dataset_iterationtype_forloop_mode_eager(api_type='wrap_into_dataset', distribution=MultiWorkerMirrored2x1CPU, drop_remainder=False, input_type='dataset', iteration_type='for_loop', mode='eager') ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 580, in decorator results = runner.run(_test_runner, args=(test_id, _env)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/input_lib_test_cpu.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 985, in run conn.send((fn, args or [], kwargs or {})) File "/usr/lib/python3.11/multiprocessing/connection.py", line 205, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/usr/lib/python3.11/multiprocessing/connection.py", line 410, in _send_bytes self._send(header + buf) File "/usr/lib/python3.11/multiprocessing/connection.py", line 367, in _send n = write(self._handle, buf) ^^^^^^^^^^^^^^^^^^^^^^^^ BrokenPipeError: [Errno 32] Broken pipe ---------------------------------------------------------------------- Ran 72 tests in 14.267s FAILED (errors=6, skipped=51) 2023-05-14 22:08:52.033922: I tensorflow/core/data/service/server_lib.cc:94] Shut down DispatchServer server running at port 40477 2023-05-14 22:08:52.034255: I tensorflow/core/data/service/server_lib.cc:94] Shut down WorkerServer server running at port 38321 2023-05-14 22:08:52.034487: I tensorflow/core/data/service/server_lib.cc:94] Shut down WorkerServer server running at port 39287 2023-05-14 22:08:52.034782: I tensorflow/core/data/service/server_lib.cc:94] Shut down WorkerServer server running at port 35821 2023-05-14 22:08:52.035426: I tensorflow/core/data/service/server_lib.cc:94] Shut down DispatchServer server running at port 35283 2023-05-14 22:08:52.035853: I tensorflow/core/data/service/server_lib.cc:94] Shut down WorkerServer server running at port 37201 2023-05-14 22:08:52.036277: I tensorflow/core/data/service/server_lib.cc:94] Shut down WorkerServer server running at port 34347 2023-05-14 22:08:52.036815: I tensorflow/core/data/service/server_lib.cc:94] Shut down WorkerServer server running at port 45989 ================================================================================ ==================== Test output for //tensorflow/dtensor/python/tests:input_util_test (shard 1 of 8): Running tests under Python 3.11.3: /usr/local/bin/python3 [ RUN ] DTensorDatasetTest.testForInIterationEager I0514 22:23:09.432196 281473212314496 mesh_util.py:35] This is client 0 of 1 clients I0514 22:23:09.432419 281473212314496 mesh_util.py:36] Number of global CPU devices: 16 I0514 22:23:09.432580 281473212314496 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0514 22:23:09.433131 281473212314496 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0514 22:23:09.433325 281473212314496 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-05-14 22:23:09.941389: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_0}}]] 2023-05-14 22:23:10.117306: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:10.172243: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:10.249863: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:10.298667: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:10.336712: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:10.337446: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:10.413505: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:10.414267: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:10.470813: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:10.471597: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:10.530048: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:10.530811: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:10.600900: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:10.601640: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:10.656485: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:10.657288: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:10.745830: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:10.746592: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:10.803307: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:10.804090: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:10.857362: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:10.858054: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:10.912547: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:10.913275: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:10.980043: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:10.980798: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:11.049759: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:11.050545: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:11.121013: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:11.121809: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:11.177283: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:11.178046: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:11.233078: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:11.233816: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:11.290351: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:11.291098: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:11.542897: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:8] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Operation was cancelled [[{{node tf.StatefulPartitionedCall/eager_operation}}]] 2023-05-14 22:23:11.542997: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:3] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Operation was cancelled [[{{node tf.StatefulPartitionedCall/eager_operation}}]] 2023-05-14 22:23:11.556214: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:4] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Operation was cancelled [[{{node tf.StatefulPartitionedCall/eager_operation}}]] 2023-05-14 22:23:11.556330: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:1] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Operation was cancelled [[{{node tf.StatefulPartitionedCall/eager_operation}}]] 2023-05-14 22:23:11.559709: E tensorflow/dtensor/cc/dtensor_device.cc:2212] Encountered error while executing function: CopyToMesh__func_16557393514391924093_5950845106898307211_6858324592425701659_1 for mesh : |batch=4,height=2,width=2|0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15|0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15|/job:localhost/replica:0/task:0/device:CPU:0,/job:localhost/replica:0/task:0/device:CPU:1,/job:localhost/replica:0/task:0/device:CPU:2,/job:localhost/replica:0/task:0/device:CPU:3,/job:localhost/replica:0/task:0/device:CPU:4,/job:localhost/replica:0/task:0/device:CPU:5,/job:localhost/replica:0/task:0/device:CPU:6,/job:localhost/replica:0/task:0/device:CPU:7,/job:localhost/replica:0/task:0/device:CPU:8,/job:localhost/replica:0/task:0/device:CPU:9,/job:localhost/replica:0/task:0/device:CPU:10,/job:localhost/replica:0/task:0/device:CPU:11,/job:localhost/replica:0/task:0/device:CPU:12,/job:localhost/replica:0/task:0/device:CPU:13,/job:localhost/replica:0/task:0/device:CPU:14,/job:localhost/replica:0/task:0/device:CPU:15 / error : {{function_node IteratorGetNext__func_2875885119105152429_12989052755378955192_1628789561242421788_0}} End of sequence [[{{node tf.StatefulPartitionedCall/eager_operation}}]] Encountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors. 2023-05-14 22:23:11.559907: E tensorflow/dtensor/cc/dtensor_device.cc:2246] Error executing CopyToMesh {{function_node IteratorGetNext__func_2875885119105152429_12989052755378955192_1628789561242421788_0}} End of sequence [[{{node tf.StatefulPartitionedCall/eager_operation}}]] Encountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors. [ FAILED ] DTensorDatasetTest.testForInIterationEager INFO:tensorflow:time(__main__.DTensorDatasetTest.testForInIterationEager): 2.21s I0514 22:23:11.645339 281473212314496 test_util.py:2464] time(__main__.DTensorDatasetTest.testForInIterationEager): 2.21s [ RUN ] DTensorDatasetTest.testIterOnBatchedDatasetGraph I0514 22:23:11.646415 281473212314496 mesh_util.py:35] This is client 0 of 1 clients I0514 22:23:11.646590 281473212314496 mesh_util.py:36] Number of global CPU devices: 16 I0514 22:23:11.646769 281473212314496 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0514 22:23:11.647173 281473212314496 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0514 22:23:11.647295 281473212314496 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-05-14 22:23:11.864164: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:11.925221: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:12.011310: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:12.083778: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:12.133841: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:12.134701: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:12.260563: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:12.261437: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:12.391588: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:12.392496: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:12.473249: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:12.474173: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:12.574570: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:12.575470: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:12.665293: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:12.666257: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:12.741053: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:12.742016: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:13.263855: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:13.264815: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:13.381892: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:13.382837: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:13.448671: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:13.449609: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:13.517953: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:13.518926: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:13.599881: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:13.600806: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:13.690513: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:13.691421: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:13.781735: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:13.782677: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:13.859989: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:13.860943: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:13.939478: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:13.940418: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:14.058821: I tensorflow/dtensor/cc/dtensor_device.cc:1574] DTensor cache key lookup missed for __inference_train_619. DTensor is (re-)computing its SPMD transformation. 2023-05-14 22:23:14.066569: I tensorflow/dtensor/cc/dtensor_device.cc:1643] DTensor cache key lookup missed for __inference_train_619. DTensor is (re-)computing its ExecutionFunctions. 2023-05-14 22:23:14.174741: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_1' with dtype float and shape [1] [[{{node Placeholder/_1}}]] INFO:tensorflow:time(__main__.DTensorDatasetTest.testIterOnBatchedDatasetGraph): 3.17s I0514 22:23:14.813019 281473212314496 test_util.py:2464] time(__main__.DTensorDatasetTest.testIterOnBatchedDatasetGraph): 3.17s [ OK ] DTensorDatasetTest.testIterOnBatchedDatasetGraph [ RUN ] DTensorDatasetTest.testIterWithLayouts1 (images_sharding=['unsharded', 'unsharded', 'unsharded', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=False, through_dtensor=True) I0514 22:23:14.814609 281473212314496 mesh_util.py:35] This is client 0 of 1 clients I0514 22:23:14.814750 281473212314496 mesh_util.py:36] Number of global CPU devices: 16 I0514 22:23:14.814886 281473212314496 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0514 22:23:14.815477 281473212314496 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0514 22:23:14.815603 281473212314496 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-05-14 22:23:15.139431: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:15.227159: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:15.228119: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:15.305375: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:15.306359: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:15.376876: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:15.377864: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:15.447970: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:15.448996: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:15.529359: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:15.530301: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:15.603174: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:15.604141: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:15.672949: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:15.673924: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:15.803291: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:15.804247: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:15.905568: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:15.906637: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:15.975086: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:15.976029: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:16.045520: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:16.046514: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:16.112303: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:16.113229: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:16.178999: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:16.179924: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:16.279460: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:16.280383: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:16.351395: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:16.352300: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:16.428568: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:16.429471: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:16.739026: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_1' with dtype float and shape [1] [[{{node Placeholder/_1}}]] INFO:tensorflow:time(__main__.DTensorDatasetTest.testIterWithLayouts1 (images_sharding=['unsharded', 'unsharded', 'unsharded', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=False, through_dtensor=True)): 2.38s I0514 22:23:17.196082 281473212314496 test_util.py:2464] time(__main__.DTensorDatasetTest.testIterWithLayouts1 (images_sharding=['unsharded', 'unsharded', 'unsharded', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=False, through_dtensor=True)): 2.38s [ OK ] DTensorDatasetTest.testIterWithLayouts1 (images_sharding=['unsharded', 'unsharded', 'unsharded', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=False, through_dtensor=True) [ RUN ] DTensorDatasetTest.testIterWithLayouts17 (images_sharding=['batch', 'height', 'width', 'unsharded'], labels_sharding=['batch', 'unsharded'], is_graph=False, through_dtensor=True) I0514 22:23:17.197700 281473212314496 mesh_util.py:35] This is client 0 of 1 clients I0514 22:23:17.197866 281473212314496 mesh_util.py:36] Number of global CPU devices: 16 I0514 22:23:17.198002 281473212314496 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0514 22:23:17.198487 281473212314496 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0514 22:23:17.198621 281473212314496 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-05-14 22:23:17.508243: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:17.587865: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:17.657940: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:17.721962: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:17.794884: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:17.795912: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:17.882398: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:17.883329: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:17.950056: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:17.951072: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:18.021513: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:18.022489: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:18.090357: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:18.091300: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:18.181443: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:18.182389: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:18.263588: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:18.264537: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:18.331487: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:18.332421: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:18.401987: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:18.402978: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:18.470393: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:18.471362: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:18.541179: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:18.542117: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:18.618242: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:18.619204: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:18.686827: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:18.687791: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:18.751977: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:18.752869: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:18.814552: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:18.815493: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:18.878957: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:18.879848: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:19.051887: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_1' with dtype float and shape [1] [[{{node Placeholder/_1}}]] INFO:tensorflow:time(__main__.DTensorDatasetTest.testIterWithLayouts17 (images_sharding=['batch', 'height', 'width', 'unsharded'], labels_sharding=['batch', 'unsharded'], is_graph=False, through_dtensor=True)): 2.17s I0514 22:23:19.371238 281473212314496 test_util.py:2464] time(__main__.DTensorDatasetTest.testIterWithLayouts17 (images_sharding=['batch', 'height', 'width', 'unsharded'], labels_sharding=['batch', 'unsharded'], is_graph=False, through_dtensor=True)): 2.17s [ OK ] DTensorDatasetTest.testIterWithLayouts17 (images_sharding=['batch', 'height', 'width', 'unsharded'], labels_sharding=['batch', 'unsharded'], is_graph=False, through_dtensor=True) [ RUN ] DTensorDatasetTest.testIterWithLayouts3 (images_sharding=['unsharded', 'unsharded', 'unsharded', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=True, through_dtensor=True) I0514 22:23:19.373176 281473212314496 mesh_util.py:35] This is client 0 of 1 clients I0514 22:23:19.373380 281473212314496 mesh_util.py:36] Number of global CPU devices: 16 I0514 22:23:19.373526 281473212314496 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0514 22:23:19.373989 281473212314496 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0514 22:23:19.374125 281473212314496 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-05-14 22:23:19.685069: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:19.811436: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:19.812353: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:19.883770: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:19.884674: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:19.953534: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:19.954464: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:20.024233: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:20.025189: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:20.095085: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:20.096049: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:20.166265: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:20.167245: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:20.237088: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:20.238073: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:20.352271: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:20.353289: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:20.429403: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:20.430372: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:20.500576: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:20.501507: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:20.581447: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:20.582387: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:20.653219: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:20.654180: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:20.728752: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:20.729668: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:20.796977: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:20.797896: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:20.865227: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:20.866300: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:20.950579: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:20.951515: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:21.063124: I tensorflow/dtensor/cc/dtensor_device.cc:1574] DTensor cache key lookup missed for __inference_train_1463. DTensor is (re-)computing its SPMD transformation. 2023-05-14 22:23:21.070688: I tensorflow/dtensor/cc/dtensor_device.cc:1643] DTensor cache key lookup missed for __inference_train_1463. DTensor is (re-)computing its ExecutionFunctions. 2023-05-14 22:23:21.201120: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_1' with dtype float and shape [1] [[{{node Placeholder/_1}}]] INFO:tensorflow:time(__main__.DTensorDatasetTest.testIterWithLayouts3 (images_sharding=['unsharded', 'unsharded', 'unsharded', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=True, through_dtensor=True)): 2.55s I0514 22:23:21.923657 281473212314496 test_util.py:2464] time(__main__.DTensorDatasetTest.testIterWithLayouts3 (images_sharding=['unsharded', 'unsharded', 'unsharded', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=True, through_dtensor=True)): 2.55s [ OK ] DTensorDatasetTest.testIterWithLayouts3 (images_sharding=['unsharded', 'unsharded', 'unsharded', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=True, through_dtensor=True) [ RUN ] DTensorDatasetTest.testMixedLayoutsFails I0514 22:23:21.925024 281473212314496 mesh_util.py:35] This is client 0 of 1 clients I0514 22:23:21.925142 281473212314496 mesh_util.py:36] Number of global CPU devices: 16 I0514 22:23:21.925271 281473212314496 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0514 22:23:21.925660 281473212314496 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0514 22:23:21.925781 281473212314496 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] INFO:tensorflow:time(__main__.DTensorDatasetTest.testMixedLayoutsFails): 0.43s I0514 22:23:22.352983 281473212314496 test_util.py:2464] time(__main__.DTensorDatasetTest.testMixedLayoutsFails): 0.43s [ OK ] DTensorDatasetTest.testMixedLayoutsFails [ RUN ] InputUtilHelpersTest.testShardCounts1 (mesh_dims=[('batch', 8)], layout_specs=['batch'], batch_dim=None, counts=[8]) I0514 22:23:22.354296 281473212314496 mesh_util.py:35] This is client 0 of 1 clients I0514 22:23:22.354426 281473212314496 mesh_util.py:36] Number of global CPU devices: 8 I0514 22:23:22.354556 281473212314496 mesh_util.py:39] Global device IDs: [0 1 2 3 4 5 6 7] I0514 22:23:22.354862 281473212314496 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7] I0514 22:23:22.354979 281473212314496 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7'] INFO:tensorflow:time(__main__.InputUtilHelpersTest.testShardCounts1 (mesh_dims=[('batch', 8)], layout_specs=['batch'], batch_dim=None, counts=[8])): 0.0s I0514 22:23:22.355414 281473212314496 test_util.py:2464] time(__main__.InputUtilHelpersTest.testShardCounts1 (mesh_dims=[('batch', 8)], layout_specs=['batch'], batch_dim=None, counts=[8])): 0.0s [ OK ] InputUtilHelpersTest.testShardCounts1 (mesh_dims=[('batch', 8)], layout_specs=['batch'], batch_dim=None, counts=[8]) ====================================================================== ERROR: testForInIterationEager (__main__.DTensorDatasetTest) DTensorDatasetTest.testForInIterationEager testForInIterationEager(False) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/dtensor/python/tests/input_util_test.py", line 188, in testForInIteration self.assertDTensorEqual(output, images_layout, d_output) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/dtensor/python/tests/test_util.py", line 303, in assertDTensorEqual unpacked = [t.numpy() for t in api.unpack(result_dtensor)] ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/dtensor/python/api.py", line 356, in unpack return _dtensor_device().unpack(tensor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/dtensor/python/dtensor_device.py", line 246, in unpack raise core._status_to_exception(e) from None # pylint: disable=protected-access ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tensorflow.python.framework.errors_impl.CancelledError: {{function_node AddV2__func_8542498174902015580_13230101824050524202_2386512056599665587_2}} Function was cancelled before it was started ---------------------------------------------------------------------- Ran 7 tests in 12.925s FAILED (errors=1) ================================================================================ ==================== Test output for //tensorflow/dtensor/python/tests:input_util_test (shard 1 of 8): Running tests under Python 3.11.3: /usr/local/bin/python3 [ RUN ] DTensorDatasetTest.testForInIterationEager I0514 22:23:26.362349 281473058042752 mesh_util.py:35] This is client 0 of 1 clients I0514 22:23:26.362576 281473058042752 mesh_util.py:36] Number of global CPU devices: 16 I0514 22:23:26.362747 281473058042752 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0514 22:23:26.363317 281473058042752 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0514 22:23:26.363511 281473058042752 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-05-14 22:23:26.755038: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_0}}]] 2023-05-14 22:23:26.888009: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:26.951121: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:27.009704: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:27.066689: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:27.109342: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:27.110088: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:27.387245: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:27.388022: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:27.508260: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:27.509035: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:27.608454: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:27.609256: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:27.797881: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:27.798652: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:27.870855: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:27.871618: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:27.963886: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:27.964684: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:28.082480: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:28.083258: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:28.227212: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:28.227980: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:28.310652: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:28.311379: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:28.441851: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:28.442623: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:28.538568: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:28.539409: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:28.624139: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:28.624968: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:28.697949: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:28.698812: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:28.759900: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:28.760701: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:28.877192: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:28.877996: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-05-14 22:23:29.341532: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:10] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Operation was cancelled [[{{node tf.StatefulPartitionedCall/eager_operation}}]] 2023-05-14 22:23:29.341635: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:7] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Operation was cancelled [[{{node tf.StatefulPartitionedCall/eager_operation}}]] 2023-05-14 22:23:29.343415: E tensorflow/dtensor/cc/dtensor_device.cc:2212] Encountered error while executing function: CopyToMesh__func_9532756592597434163_5950845106898307211_6858324592425701659_1 for mesh : |batch=4,height=2,width=2|0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15|0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15|/job:localhost/replica:0/task:0/device:CPU:0,/job:localhost/replica:0/task:0/device:CPU:1,/job:localhost/replica:0/task:0/device:CPU:2,/job:localhost/replica:0/task:0/device:CPU:3,/job:localhost/replica:0/task:0/device:CPU:4,/job:localhost/replica:0/task:0/device:CPU:5,/job:localhost/replica:0/task:0/device:CPU:6,/job:localhost/replica:0/task:0/device:CPU:7,/job:localhost/replica:0/task:0/device:CPU:8,/job:localhost/replica:0/task:0/device:CPU:9,/job:localhost/replica:0/task:0/device:CPU:10,/job:localhost/replica:0/task:0/device:CPU:11,/job:localhost/replica:0/task:0/device:CPU:12,/job:localhost/replica:0/task:0/device:CPU:13,/job:localhost/replica:0/task:0/device:CPU:14,/job:localhost/replica:0/task:0/device:CPU:15 / error : {{function_node IteratorGetNext__func_11512966264490218070_12989052755378955192_1628789561242421788_0}} End of sequence [[{{node tf.StatefulPartitionedCall/eager_operation}}]] Encountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors. 2023-05-14 22:23:29.356227: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:5] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Operation was cancelled [[{{node tf.StatefulPartitionedCall/eager_operation}}]] 2023-05-14 22:23:29.364534: E tensorflow/dtensor/cc/dtensor_device.cc:2246] Error executing CopyToMesh {{function_node IteratorGetNext__func_11512966264490218070_12989052755378955192_1628789561242421788_0}} End of sequence [[{{node tf.StatefulPartitionedCall/eager_operation}}]] Encountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors. [ FAILED ] DTensorDatasetTest.testForInIterationEager INFO:tensorflow:time(__main__.DTensorDatasetTest.testForInIterationEager): 3.1s I0514 22:23:29.461348 281473058042752 test_util.py:2464] time(__main__.DTensorDatasetTest.testForInIterationEager): 3.1s [ RUN ] DTensorDatasetTest.testIterOnBatchedDatasetGraph I0514 22:23:29.462762 281473058042752 mesh_util.py:35] This is client 0 of 1 clients I0514 22:23:29.462965 281473058042752 mesh_util.py:36] Number of global CPU devices: 16 I0514 22:23:29.463148 281473058042752 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0514 22:23:29.463598 281473058042752 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0514 22:23:29.463723 281473058042752 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-05-14 22:23:29.657012: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:29.723503: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:29.787646: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:29.850124: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:29.902890: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:29.903785: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:29.975328: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:29.981350: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:30.148877: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:30.149801: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:30.308336: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:30.309274: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:30.390037: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:30.390937: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:30.459784: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:30.460689: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:30.526699: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:30.527676: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:30.648919: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:30.649852: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:30.732371: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:30.733308: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:30.830970: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:30.831896: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:30.937922: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:30.938868: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:31.074925: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:31.075932: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:31.201256: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:31.202232: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:31.287751: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:31.288694: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:31.370561: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:31.371530: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:31.433881: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:31.434800: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:31.542800: I tensorflow/dtensor/cc/dtensor_device.cc:1574] DTensor cache key lookup missed for __inference_train_619. DTensor is (re-)computing its SPMD transformation. 2023-05-14 22:23:31.550160: I tensorflow/dtensor/cc/dtensor_device.cc:1643] DTensor cache key lookup missed for __inference_train_619. DTensor is (re-)computing its ExecutionFunctions. 2023-05-14 22:23:31.667419: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_1' with dtype float and shape [1] [[{{node Placeholder/_1}}]] INFO:tensorflow:time(__main__.DTensorDatasetTest.testIterOnBatchedDatasetGraph): 2.86s I0514 22:23:32.319566 281473058042752 test_util.py:2464] time(__main__.DTensorDatasetTest.testIterOnBatchedDatasetGraph): 2.86s [ OK ] DTensorDatasetTest.testIterOnBatchedDatasetGraph [ RUN ] DTensorDatasetTest.testIterWithLayouts1 (images_sharding=['unsharded', 'unsharded', 'unsharded', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=False, through_dtensor=True) I0514 22:23:32.320923 281473058042752 mesh_util.py:35] This is client 0 of 1 clients I0514 22:23:32.321048 281473058042752 mesh_util.py:36] Number of global CPU devices: 16 I0514 22:23:32.321180 281473058042752 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0514 22:23:32.321590 281473058042752 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0514 22:23:32.321713 281473058042752 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-05-14 22:23:32.573352: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:32.626061: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:32.627059: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:32.724134: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:32.725061: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:32.795139: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:32.796165: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:32.866929: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:32.867949: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:32.945937: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:32.946939: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:33.018375: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:33.019292: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:33.149307: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:33.150227: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:33.216581: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:33.217538: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:33.342049: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:33.342984: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:33.432537: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:33.433442: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:33.508584: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:33.509531: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:33.608670: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:33.609582: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:33.689982: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:33.690921: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:33.768514: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:33.769414: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:33.849677: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:33.850616: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:34.026527: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:34.027462: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:34.379490: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_1' with dtype float and shape [1] [[{{node Placeholder/_1}}]] INFO:tensorflow:time(__main__.DTensorDatasetTest.testIterWithLayouts1 (images_sharding=['unsharded', 'unsharded', 'unsharded', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=False, through_dtensor=True)): 2.56s I0514 22:23:34.881108 281473058042752 test_util.py:2464] time(__main__.DTensorDatasetTest.testIterWithLayouts1 (images_sharding=['unsharded', 'unsharded', 'unsharded', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=False, through_dtensor=True)): 2.56s [ OK ] DTensorDatasetTest.testIterWithLayouts1 (images_sharding=['unsharded', 'unsharded', 'unsharded', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=False, through_dtensor=True) [ RUN ] DTensorDatasetTest.testIterWithLayouts17 (images_sharding=['batch', 'height', 'width', 'unsharded'], labels_sharding=['batch', 'unsharded'], is_graph=False, through_dtensor=True) I0514 22:23:34.882582 281473058042752 mesh_util.py:35] This is client 0 of 1 clients I0514 22:23:34.882704 281473058042752 mesh_util.py:36] Number of global CPU devices: 16 I0514 22:23:34.882833 281473058042752 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0514 22:23:34.883263 281473058042752 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0514 22:23:34.883385 281473058042752 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-05-14 22:23:35.333815: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:35.758088: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:35.866452: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:35.969703: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:36.135845: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:36.136792: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:36.251000: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:36.251868: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:36.446492: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:36.447603: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:36.612152: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:36.613102: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:36.838612: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:36.839512: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:37.012046: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:37.012946: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:37.151916: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:37.152847: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:37.579793: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:37.580701: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:37.791855: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:37.792794: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:37.957401: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:37.958351: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:38.106329: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:38.107274: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:38.321824: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:38.322763: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:38.597300: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:38.598265: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:38.931257: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:38.932182: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:39.113882: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:39.114836: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:39.290536: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:39.291455: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:39.700019: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_1' with dtype float and shape [1] [[{{node Placeholder/_1}}]] INFO:tensorflow:time(__main__.DTensorDatasetTest.testIterWithLayouts17 (images_sharding=['batch', 'height', 'width', 'unsharded'], labels_sharding=['batch', 'unsharded'], is_graph=False, through_dtensor=True)): 5.75s I0514 22:23:40.630361 281473058042752 test_util.py:2464] time(__main__.DTensorDatasetTest.testIterWithLayouts17 (images_sharding=['batch', 'height', 'width', 'unsharded'], labels_sharding=['batch', 'unsharded'], is_graph=False, through_dtensor=True)): 5.75s [ OK ] DTensorDatasetTest.testIterWithLayouts17 (images_sharding=['batch', 'height', 'width', 'unsharded'], labels_sharding=['batch', 'unsharded'], is_graph=False, through_dtensor=True) [ RUN ] DTensorDatasetTest.testIterWithLayouts3 (images_sharding=['unsharded', 'unsharded', 'unsharded', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=True, through_dtensor=True) I0514 22:23:40.631796 281473058042752 mesh_util.py:35] This is client 0 of 1 clients I0514 22:23:40.631916 281473058042752 mesh_util.py:36] Number of global CPU devices: 16 I0514 22:23:40.632046 281473058042752 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0514 22:23:40.632438 281473058042752 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0514 22:23:40.632558 281473058042752 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-05-14 22:23:41.239438: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-05-14 22:23:41.372330: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:41.373220: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:41.611794: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:41.612687: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:41.733723: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:41.734650: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:41.890571: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:41.891499: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:42.033062: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:42.033996: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:42.479904: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:42.480818: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:42.630458: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:42.631372: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:42.771201: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:42.772128: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:42.961453: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:42.962385: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:43.380168: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:43.381084: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:43.701028: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:43.701945: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:43.837720: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:43.838634: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:43.998616: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:43.999502: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:44.218545: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:44.219456: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-05-14 22:23:44.381601: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:44.382491: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-05-14 22:23:44.461676: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-05-14 22:23:44.462577: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-05-14 22:23:44.834512: I tensorflow/dtensor/cc/dtensor_device.cc:1574] DTensor cache key lookup missed for __inference_train_1463. DTensor is (re-)computing its SPMD transformation. 2023-05-14 22:23:44.842020: I tensorflow/dtensor/cc/dtensor_device.cc:1643] DTensor cache key lookup missed for __inference_train_1463. DTensor is (re-)computing its ExecutionFunctions. 2023-05-14 22:23:45.021464: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_1' with dtype float and shape [1] [[{{node Placeholder/_1}}]] INFO:tensorflow:time(__main__.DTensorDatasetTest.testIterWithLayouts3 (images_sharding=['unsharded', 'unsharded', 'unsharded', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=True, through_dtensor=True)): 5.74s I0514 22:23:46.368798 281473058042752 test_util.py:2464] time(__main__.DTensorDatasetTest.testIterWithLayouts3 (images_sharding=['unsharded', 'unsharded', 'unsharded', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=True, through_dtensor=True)): 5.74s [ OK ] DTensorDatasetTest.testIterWithLayouts3 (images_sharding=['unsharded', 'unsharded', 'unsharded', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=True, through_dtensor=True) [ RUN ] DTensorDatasetTest.testMixedLayoutsFails I0514 22:23:46.370296 281473058042752 mesh_util.py:35] This is client 0 of 1 clients I0514 22:23:46.370417 281473058042752 mesh_util.py:36] Number of global CPU devices: 16 I0514 22:23:46.370551 281473058042752 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0514 22:23:46.370985 281473058042752 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0514 22:23:46.371108 281473058042752 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] INFO:tensorflow:time(__main__.DTensorDatasetTest.testMixedLayoutsFails): 0.73s I0514 22:23:47.098577 281473058042752 test_util.py:2464] time(__main__.DTensorDatasetTest.testMixedLayoutsFails): 0.73s [ OK ] DTensorDatasetTest.testMixedLayoutsFails [ RUN ] InputUtilHelpersTest.testShardCounts1 (mesh_dims=[('batch', 8)], layout_specs=['batch'], batch_dim=None, counts=[8]) I0514 22:23:47.100116 281473058042752 mesh_util.py:35] This is client 0 of 1 clients I0514 22:23:47.100250 281473058042752 mesh_util.py:36] Number of global CPU devices: 8 I0514 22:23:47.100381 281473058042752 mesh_util.py:39] Global device IDs: [0 1 2 3 4 5 6 7] I0514 22:23:47.100722 281473058042752 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7] I0514 22:23:47.100840 281473058042752 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7'] INFO:tensorflow:time(__main__.InputUtilHelpersTest.testShardCounts1 (mesh_dims=[('batch', 8)], layout_specs=['batch'], batch_dim=None, counts=[8])): 0.0s I0514 22:23:47.101301 281473058042752 test_util.py:2464] time(__main__.InputUtilHelpersTest.testShardCounts1 (mesh_dims=[('batch', 8)], layout_specs=['batch'], batch_dim=None, counts=[8])): 0.0s [ OK ] InputUtilHelpersTest.testShardCounts1 (mesh_dims=[('batch', 8)], layout_specs=['batch'], batch_dim=None, counts=[8]) ====================================================================== ERROR: testForInIterationEager (__main__.DTensorDatasetTest) DTensorDatasetTest.testForInIterationEager testForInIterationEager(False) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/dtensor/python/tests/input_util_test.py", line 188, in testForInIteration self.assertDTensorEqual(output, images_layout, d_output) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/dtensor/python/tests/test_util.py", line 303, in assertDTensorEqual unpacked = [t.numpy() for t in api.unpack(result_dtensor)] ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/dtensor/python/api.py", line 356, in unpack return _dtensor_device().unpack(tensor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/dtensor/python/dtensor_device.py", line 246, in unpack raise core._status_to_exception(e) from None # pylint: disable=protected-access ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tensorflow.python.framework.errors_impl.CancelledError: {{function_node AddV2__func_6686991105081273317_13230101824050524202_2386512056599665587_2}} Function was cancelled before it was started ---------------------------------------------------------------------- Ran 7 tests in 20.741s FAILED (errors=1) ================================================================================ //tensorflow/c:c_api_experimental_test PASSED in 37.9s //tensorflow/c:c_api_function_test PASSED in 33.7s //tensorflow/c:c_api_test_cpu PASSED in 40.2s //tensorflow/c:c_test PASSED in 32.6s //tensorflow/c:env_test_cpu PASSED in 29.5s //tensorflow/c:kernels_test_cpu PASSED in 43.0s //tensorflow/c:ops_test PASSED in 29.9s //tensorflow/c:while_loop_test PASSED in 41.3s //tensorflow/c/eager:c_api_cluster_test_cpu PASSED in 36.4s //tensorflow/c/eager:c_api_remote_function_test_cpu PASSED in 41.1s //tensorflow/c/eager:c_api_remote_test_cpu PASSED in 42.1s //tensorflow/c/eager:c_api_test_cpu PASSED in 41.7s //tensorflow/c/eager:custom_device_test PASSED in 34.2s //tensorflow/c/eager/parallel_device:parallel_device_lib_test PASSED in 40.8s //tensorflow/c/eager/parallel_device:parallel_device_remote_test PASSED in 37.1s //tensorflow/c/eager/parallel_device:parallel_device_test PASSED in 36.1s //tensorflow/c/experimental/filesystem/plugins/gcs:expiring_lru_cache_test PASSED in 0.1s //tensorflow/c/experimental/filesystem/plugins/gcs:ram_file_block_cache_test PASSED in 2.4s //tensorflow/c/experimental/grappler:grappler_test PASSED in 28.9s //tensorflow/c/experimental/ops/gen/common:case_format_test PASSED in 0.6s //tensorflow/c/experimental/ops/gen/cpp:cpp_generator_test PASSED in 0.9s //tensorflow/c/experimental/ops/gen/cpp/renderers:renderer_test PASSED in 0.8s //tensorflow/c/experimental/saved_model/core:constant_loading_test PASSED in 23.6s //tensorflow/c/experimental/saved_model/core:object_graph_traversal_test PASSED in 15.8s //tensorflow/c/experimental/saved_model/core:saved_variable_loading_test PASSED in 22.7s //tensorflow/c/experimental/saved_model/core:signature_flattening_test PASSED in 13.2s //tensorflow/c/experimental/saved_model/core:tf_concrete_function_loading_test PASSED in 12.0s //tensorflow/c/experimental/saved_model/core/ops:restore_ops_test PASSED in 15.7s //tensorflow/c/experimental/saved_model/core/ops:variable_ops_test PASSED in 16.2s //tensorflow/c/experimental/saved_model/internal:saved_model_api_test PASSED in 38.3s //tensorflow/c/experimental/stream_executor:stream_executor_test PASSED in 0.1s //tensorflow/c/kernels:bitcast_op_test PASSED in 0.8s //tensorflow/c/kernels:summary_op_benchmark_test PASSED in 1.1s //tensorflow/c/kernels:summary_op_test PASSED in 1.5s //tensorflow/c/kernels:tensor_shape_utils_test PASSED in 0.2s //tensorflow/cc:cc_op_gen_test PASSED in 0.1s //tensorflow/cc:client_client_session_test PASSED in 2.3s //tensorflow/cc:coordinator_test PASSED in 4.8s //tensorflow/cc:framework_cc_ops_test PASSED in 4.0s //tensorflow/cc:framework_gradient_checker_test PASSED in 3.6s //tensorflow/cc:framework_gradients_test PASSED in 4.6s //tensorflow/cc:framework_scope_test PASSED in 0.8s //tensorflow/cc:framework_while_gradients_test PASSED in 2.5s //tensorflow/cc:gradients_array_grad_test PASSED in 5.4s //tensorflow/cc:gradients_data_flow_grad_test PASSED in 1.9s //tensorflow/cc:gradients_functional_grad_test PASSED in 2.0s //tensorflow/cc:gradients_image_grad_test PASSED in 6.8s //tensorflow/cc:gradients_linalg_grad_test PASSED in 2.8s //tensorflow/cc:gradients_manip_grad_test PASSED in 2.9s //tensorflow/cc:gradients_math_grad_test PASSED in 6.0s //tensorflow/cc:gradients_nn_grad_test PASSED in 2.7s //tensorflow/cc:gradients_resource_variable_grad_test PASSED in 2.7s //tensorflow/cc:ops_const_op_test PASSED in 0.8s //tensorflow/cc:ops_while_loop_test PASSED in 2.7s //tensorflow/cc:queue_runner_test PASSED in 13.3s //tensorflow/cc/experimental/base/tests:tensor_test PASSED in 0.2s //tensorflow/cc/experimental/base/tests:tensorhandle_test PASSED in 38.0s //tensorflow/cc/experimental/libexport:load_test PASSED in 0.5s //tensorflow/cc/experimental/libexport:save_test PASSED in 0.3s //tensorflow/cc/experimental/libtf:libtf_module_test PASSED in 37.4s //tensorflow/cc/experimental/libtf:libtf_object_test PASSED in 0.1s //tensorflow/cc/experimental/libtf:libtf_perf_test PASSED in 0.8s //tensorflow/cc/experimental/libtf:libtf_runtime_test PASSED in 34.1s //tensorflow/cc/experimental/libtf:libtf_transform_test PASSED in 38.0s //tensorflow/cc/experimental/libtf:libtf_value_test PASSED in 0.2s //tensorflow/cc/experimental/libtf:libtf_visit_test PASSED in 0.1s //tensorflow/cc/experimental/libtf/impl:iostream_test PASSED in 0.1s //tensorflow/cc/experimental/libtf/impl:none_test PASSED in 0.1s //tensorflow/cc/experimental/libtf/impl:scalars_test PASSED in 0.3s //tensorflow/cc/experimental/libtf/impl:string_test PASSED in 0.6s //tensorflow/cc/experimental/libtf/impl:tensor_spec_test PASSED in 0.9s //tensorflow/cc/saved_model:bundle_v2_test PASSED in 0.1s //tensorflow/cc/saved_model:fingerprinting_test PASSED in 1.5s //tensorflow/cc/saved_model:metrics_test PASSED in 0.2s //tensorflow/cc/saved_model:reader_test PASSED in 0.4s //tensorflow/cc/saved_model:saved_model_bundle_lite_test PASSED in 7.4s //tensorflow/cc/saved_model:saved_model_bundle_test PASSED in 8.7s //tensorflow/cc/saved_model:util_test PASSED in 0.3s //tensorflow/cc/saved_model/experimental/tests:saved_model_api_test PASSED in 40.1s //tensorflow/cc/tools:freeze_saved_model_test PASSED in 2.1s //tensorflow/compiler/aot:codegen_test PASSED in 36.7s //tensorflow/compiler/jit:compilability_check_util_test PASSED in 29.9s //tensorflow/compiler/jit:deadness_analysis_test PASSED in 12.3s //tensorflow/compiler/jit:device_compilation_cache_test PASSED in 6.6s //tensorflow/compiler/jit:device_compilation_cluster_signature_test PASSED in 7.1s //tensorflow/compiler/jit:device_compilation_profiler_test PASSED in 27.6s //tensorflow/compiler/jit:device_compiler_client_test PASSED in 6.7s //tensorflow/compiler/jit:device_compiler_disable_test PASSED in 31.0s //tensorflow/compiler/jit:device_executable_persistor_test PASSED in 31.3s //tensorflow/compiler/jit:device_util_test PASSED in 9.5s //tensorflow/compiler/jit:encapsulate_util_test PASSED in 1.1s //tensorflow/compiler/jit:node_matchers_test PASSED in 1.0s //tensorflow/compiler/jit:resource_operation_safety_analysis_test PASSED in 12.9s //tensorflow/compiler/jit:shape_inference_test PASSED in 1.1s //tensorflow/compiler/jit:xla_activity_listener_test PASSED in 32.5s //tensorflow/compiler/jit:xla_cluster_util_test PASSED in 13.5s //tensorflow/compiler/jit:xla_compile_util_test PASSED in 5.8s //tensorflow/compiler/jit:xla_kernel_creator_test PASSED in 12.9s //tensorflow/compiler/jit:xla_launch_util_test PASSED in 26.1s //tensorflow/compiler/jit/tests:auto_clustering_test PASSED in 33.3s //tensorflow/compiler/mlir:mlir_graph_optimization_pass_test PASSED in 14.5s //tensorflow/compiler/mlir:register_common_dialects_test PASSED in 19.6s //tensorflow/compiler/mlir/lite:lstm_utils_test PASSED in 0.8s //tensorflow/compiler/mlir/lite:perception_ops_utils_test PASSED in 0.7s //tensorflow/compiler/mlir/lite:size_utils_test PASSED in 0.4s //tensorflow/compiler/mlir/lite:tftext_utils_test PASSED in 1.1s //tensorflow/compiler/mlir/lite/experimental/remat:rematerializer_test PASSED in 1.5s //tensorflow/compiler/mlir/lite/experimental/tac:execution_metadata_exporter_test PASSED in 4.2s //tensorflow/compiler/mlir/lite/experimental/tac/tests:compute-cost.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-gpu.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-nnapi.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:fold-constants-to-subgraph.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:get-alternative-subgraph.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/experimental/tac/tests:get-op-cost.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/experimental/tac/tests:pick-subgraphs.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/experimental/tac/tests:raise-target-subgraphs.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/experimental/tac/tests:target-annotation.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:device-transform-nnapi.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:simple-graph.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/metrics:error_collector_inst_test PASSED in 0.4s //tensorflow/compiler/mlir/lite/quantization:numerical_utils_test PASSED in 0.2s //tensorflow/compiler/mlir/lite/quantization/lite:quantize_model_test PASSED in 19.4s //tensorflow/compiler/mlir/lite/quantization/lite:quantize_weights_test PASSED in 16.3s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_default.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_legacy.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant_4bit.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/quantization/tests:import_quant_stats.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/sparsity:sparsify_model_test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:fold_broadcast.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:fuse_mhlo_convolution.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-inplaceupdate.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-skip-quantization-ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tf-fb-tf.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-add.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-broadcast_in_dim.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-clamp.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-compare.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-concat.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-constant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-conv.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-dot.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-gather.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-max.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-mul.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-pad.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-reshape.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-rsqrt.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-scatter.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-sub.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-add.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-broadcast.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-clamp.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-concat.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-constant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-conv.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-max.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-mul.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-pad.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-reshape.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-rsqrt.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-sub.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-allow-tf.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-smuggle-resize.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/stablehlo/tests:optimize.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-clamp.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-concat.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-conv.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-division.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-logistic.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-multiply.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-reduce-window.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-resize-bilinear.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-subtract.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-tf-quantize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:unfuse_mhlo_batch_norm.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:analyze-variables.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:canonicalize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:const-fold.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:decompose-hybrid-quantization.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:default_quant_params.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:dilated-conv.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:fuse-tftext.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:get-arithmetic-count.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:guarantee_func_has_one_use.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:inlining.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:insert_call_once_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:legalize-tf-assert.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:legalize-tf-hashtables.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:legalize-tf-no-runtime-verification.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:legalize-tf-variables.mlir.test PASSED in 2.6s //tensorflow/compiler/mlir/lite/tests:legalize-tf-while.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:legalize-tf.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/lite/tests:legalize_jax_random.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:lift_tflite_flex_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-default-to-single-batch.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-enable-dynamic-update-slice.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list.mlir.test PASSED in 5.5s //tensorflow/compiler/mlir/lite/tests:modify_io_nodes.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests:ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:optimize-after-quantization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:optimize.mlir.test PASSED in 3.9s //tensorflow/compiler/mlir/lite/tests:optimize_functional_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:optimize_no_verify.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:optimize_op_order.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:partitioned-topological-sort.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:pin-ops-with-side-effects.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:post-quantize-dynamic-range.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:post-quantize.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:prepare-composite-functions-tf.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-dynamic-range.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training-16bits.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-signed.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:prepare-quantize.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant-4bit.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:prepare-tf-with-allowing-bf16-and-f16-type-legalization.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests:prepare-tf.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:quantize-dynamic-range.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests:quantize-numeric-verify.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:quantize-variables.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:quantize.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests:raise-custom-ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:reduce_while_operands.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests:shape-inference.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:split-merged-operands.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:tfl_while_op_licm.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:tfl_while_outline.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests:trim-functions-tf.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:unfold-large-splat-constant.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.line.part.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.stack.part.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:add.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/end2end:back2back_fake_quant.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:control_flow_v1.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/end2end:conv_2d.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:conv_2d_nchw.pbtxt.test PASSED in 2.2s //tensorflow/compiler/mlir/lite/tests/end2end:custom_opdef.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/end2end:disallow_stateful_partitioned_call.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel_4bit.pbtxt.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity_4bit.pbtxt.test PASSED in 3.1s //tensorflow/compiler/mlir/lite/tests/end2end:graph-input-node.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:graph_with_placeholder_with_default.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/end2end:if_op.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:quant_stats.pbtxt.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul_disabled.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:basic_lstm.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:bucketize.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:constants.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:control_edges.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:custom_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:dynamic_shape.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:empty_input_output_names.json.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:external_constant.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:if_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:import_json.json.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_arrays.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_output_names_attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:legacy_reshape.json.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.json.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:many_attribute_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:math.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:matmul.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:multi_output_op.json.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional_input.json.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:output_arrays.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning.mlir.test PASSED in 0.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning_function_input_as_output.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quant_stats.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quantization.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:reshape.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature_with_multiple_entry_points.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:simple.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:tf_variant_type.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_function_output.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_tensor.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:while_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2exec:tfl_while_op.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:basic_lstm.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:bucketize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:custom_op_with_tflite_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d_v2.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_builtin.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_custom.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex.mlir.test PASSED in 5.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex_enable_builtin.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:dynamic_shape_constant.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fake_quant.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_exclusively.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_complex128.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_f64.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_tflite_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected.mlir.test PASSED in 0.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected_v2.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:hashtable_resource.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:if_op.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:logical.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:low_bit_packing.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_asym_attr.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_quantized.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:math.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:metadata.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v2.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v3.mlir.test PASSED in 0.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:nn.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:numeric_verify.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:optional.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:quantization.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:reshape.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_output_override.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_multiple_entry_points.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_no_inputs.mlir.test PASSED in 0.4s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_connected_control_nodes.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_unconnected_control_nodes.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf_v2.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tf_entry_function.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tfl_while_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:transpose_conv_optional.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:type_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_lstm.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_rnn.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unranked_tensor.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unsorted_segment_prod.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_func.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:while_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:calibrator_singleton_test PASSED in 0.1s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:custom_aggregator_op_test PASSED in 14.5s //tensorflow/compiler/mlir/quantization/tensorflow/cc:const_op_size_test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/cc:convert_asset_args_test PASSED in 7.1s //tensorflow/compiler/mlir/quantization/tensorflow/cc:save_variables_test PASSED in 0.5s //tensorflow/compiler/mlir/quantization/tensorflow/cc:status_macro_test PASSED in 0.3s //tensorflow/compiler/mlir/quantization/tensorflow/debugging:mlir_dump_test PASSED in 0.4s //tensorflow/compiler/mlir/quantization/tensorflow/python:concurrency_test PASSED in 39.7s //tensorflow/compiler/mlir/quantization/tensorflow/python:pywrap_quantize_model_test PASSED in 19.4s //tensorflow/compiler/mlir/quantization/tensorflow/python:representative_dataset_test PASSED in 8.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:cast_bf16_ops_to_f32.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_custom_aggregation_op_to_quant_stats.mlir.test PASSED in 3.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_fake_quant_to_qdq.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_tf_quant_ops_to_mhlo.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_tpu_model_to_cpu.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:duplicate_shape_determining_constants.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_flow.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_xla.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_custom_aggregation_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_main_function.mlir.test PASSED in 2.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_drq.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_weight_only.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_restore_op.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_save_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:issue_ids_of_custom_aggregation_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq_min_elements.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_xla.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:mark_functions_noinline.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_initializer_function_ops_to_main.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_save_function_ops_to_main.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:optimize.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_lifting.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq_per_channel.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq_per_channel.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:preprocess_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:preprocess_op_weight_only.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_drq.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_weight_only.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_xla.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_drq.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_xla.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:remove_var_init_by_const.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops_large_constants.mlir.test PASSED in 16.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:unfreeze_constants.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/quantization/tensorflow/utils:tf_to_xla_attribute_utils_test PASSED in 43.7s //tensorflow/compiler/mlir/tensorflow:bridge_logger_test PASSED in 7.5s //tensorflow/compiler/mlir/tensorflow:cluster_util_test PASSED in 0.4s //tensorflow/compiler/mlir/tensorflow:convert_tensor_test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow:convert_type_test PASSED in 0.4s //tensorflow/compiler/mlir/tensorflow:device_util_test PASSED in 0.3s //tensorflow/compiler/mlir/tensorflow:dump_graph_test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow:dump_mlir_util_test PASSED in 15.2s //tensorflow/compiler/mlir/tensorflow:error_util_test PASSED in 0.2s //tensorflow/compiler/mlir/tensorflow:tf_saved_model_test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow:tpu_rewrite_device_util_test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:add_functions_for_exported_names.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:annotate-parameter-replication.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:batchmatmul_to_einsum.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:breakup-islands.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:cannonicalize_ops_outside_compilation.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:canonicalize.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tensorflow/tests:canonicalize_compile_and_replicate_attributes.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:check_control_dependencies.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:cluster_formation.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:cluster_ops_by_policy.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:cluster_outlining.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:cluster_tf_ops_pass.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:constant-fold.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:constant_op_device_assignment.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:convert-tf-control-flow-to-scf.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:convert_control_to_data_outputs.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:convert_launch_func_to_tf_call.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:convert_session_initializer_to_function.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:convert_to_legacy_compile_and_replicate_attributes.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:decompose_reduce_dataset.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:decompose_resource_ops.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:device_assignment.mlir.test PASSED in 2.7s //tensorflow/compiler/mlir/tensorflow/tests:device_assignment_by_func_attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:device_attribute_to_launch.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:device_canonicalize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:device_copy.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:drop_while_shape_invariant.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:einsum.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:embedding_pipelining.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:empty-main.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:end-to-end-tpu-reshard-variables.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:executor_canonicalize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:executor_island_coarsening.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:executor_island_materialize_const.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:extract_head_tail_outside_compilation.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:extract_outside_compilation.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/tensorflow/tests:extract_tpu_copy_with_dynamic_shape_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:fold-broadcast.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:freeze_variables.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:func-attr-invalid.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:func-attr.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-cfg.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-regions.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:functionalize-if-fail.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:functionalize-if.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:fused_kernel_matcher.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:gpu_fusion.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:graph_pruning.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:graph_pruning_preserve_ops.mlir.test PASSED in 2.9s //tensorflow/compiler/mlir/tensorflow/tests:group_by_dialect.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:guarantee-all-funcs-one-use.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:hoist_loop_invariant.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:hoist_replicate_invariant_resource_writes.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:host_launch_to_outside_compiled.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_invalid.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_saved_model.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:inlining.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:isolate-placer.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:launch_outlining.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute_legacy.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_60.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_70.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nchw.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nhwc.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_begin.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_end.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nchw.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nhwc.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:legalize_hlo.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_arg_control_dep.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_with_control_flow.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:localize_var_handles.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program_invalid.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:lower_quantized.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:lower_tf.mlir.test PASSED in 3.4s //tensorflow/compiler/mlir/tensorflow/tests:lower_variable_ops_to_ml_program.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:mark_input_output_aliases.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:mark_ops_for_outside_compilation.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:materialize_passthrough_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:merge_control_flow.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:mlprogram.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:name_anonymous_iterators.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:optimize-arg-operand-constraint.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:optimize.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:order_by_dialect.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:outside_compiled_to_host_launch.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands_legacy.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:prepare_tpu_computation_for_tf_export.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args_functions.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:promote_var_handles_to_args.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests:readonly_references_to_resources.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:region-control-flow-to-functional.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:remove_unused_arguments.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests:remove_unused_while_results.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:replica_id_to_device_ordinal.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:replicate_invariant_op_hoisting.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:replicate_tensor_list_init_ops.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island_legacy.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:resource-alias-analysis-test.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:resource-device-inference.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:resource_analyzer.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:resource_inlining.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:resource_op_lifting.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:rewrite_tpu_embedding_ops.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:roundtrip-tf-executor.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:shape_inference.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:side-effect-analysis-test.mlir.test PASSED in 2.6s //tensorflow/compiler/mlir/tensorflow/tests:sink_constant.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:split_into_island_per_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:stack_ops_decomposition.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:strip_noinline.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests:strip_saved_module_metadata.mlir.test PASSED in 5.4s //tensorflow/compiler/mlir/tensorflow/tests:strip_tf_attributes.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tensor_array_ops_decomposition.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tensor_list_ops_decomposition.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf-executor-to-functional.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf-functional-to-executor.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tf-ops.mlir.test PASSED in 2.9s //tensorflow/compiler/mlir/tensorflow/tests:tf-reduce-identity.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_map_and_batch.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_pmap_and_batch.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_index_selector.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops_invalid.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_invalid.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_location_roundtrip.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_printer.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_side_effect.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_optimize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_deduplicate_bound_input_bindings.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_assets.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors_mutable_tensors.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init_fail.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables_invalid_session.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_mark_initialized_variables.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops_invalid.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors_interprocedural.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_remove_vars_in_session_initializer.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:tf_side_effect.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:tf_trait_folds.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu-annotate-dynamic-shape-inputs.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu-cluster-cleanup-attributes.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu-dynamic-layout-pass.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu-merge-variables-with-execute.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu-multiple-while-body-func.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu-resource-read-for-write.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:tpu-variable-runtime-reformatting.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_cluster_formation.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu_colocate_composite_resource_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_colocate_splits.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_device_propagation.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_host_computation_expansion.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_identity_pruning.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:tpu_parallel_execute_sink_resource_write.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_partitioned_op_conversion.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_reorder_replicate_and_partitioned_inputs.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tpu_resource_partitioning.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_rewrite.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests:tpu_sharding_identification.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_space_to_depth_pass.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_tail_with_tobool_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_update_embedding_enqueue_op_inputs.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:tpu_validate_inputs.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:transpose-op.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:unroll-batch-matmul.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:update_control_dependencies.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:warn_when_using_deprecated_dumps.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:while_licm.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:xla_cluster_formation.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:xla_inline_device_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:xla_rewrite.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:add.mlir.test PASSED in 2.2s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding-invalid.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding-hook.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:mlir-module-serialized-str-attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:replicate-tensor-list-init-ops.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:result-sharding.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr-invalid.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference-after-legalization.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:stablehlo_add.mlir.test PASSED in 2.3s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:executor_tpuv1_island_coarsening.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:while_op.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:executor_tpuv1_inline_tpu_island.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:while_op.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:case_op.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:executor_tpuv1_outline_tpu_island.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:while_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:add.pbtxt.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-as-fetch.pbtxt.test PASSED in 2.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-control-dep.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type-with-subtype.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type.pbtxt.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-multi-data-type-with-subtype.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-retval-attrs.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:case_op.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:const-values.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:device-arg-retval-attr.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-input-shapes.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-value-attr.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-as-fetch.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-control-dep.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:force_shared_name_for_resource_ops.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:function-func-attr.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-if-ops.pbtxt.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-while-ops.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-control-ret.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-retval-of-arg.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-custom-operation.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-default-attr.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-device-retval.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-empty-tensor-content.pbtxt.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-func-attr.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-call.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-diff-island.pbtxt.test PASSED in 2.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-same-island.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-defs.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-input-shapes.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-name-bug.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-resource-args.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-gradient-def.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-input-func-arg-name-collision.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-library.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-malformed.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-scalar-input.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-uint8-return.pbtxt.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-undefined-output.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-version-info.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-while-loop.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:invalid-output-index.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:legacy-fed-input-without-inputs.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:merge_node_with_function.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:mlir_passthrough_op.pbtxt.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multi-output-feeds.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multiple-use-next-iteration.pbtxt.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:node-locations.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes-attr.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example_v2.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:partial-device-name.pbtxt.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:prune_unused_nodes.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:quint8-const.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:shape-attrs.pbtxt.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:stateful-attribute.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:string-attr.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:switch_n.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:target.pbtxt.test PASSED in 2.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tensor-list.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tf-data-pipeline.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:unregistered_kernel.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir/batch_use_same_function:saved_model.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:aliasing_arg_attr.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:case.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:convert_tensor.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_shape_attr.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_size_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:device-arg-retval-attr.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:export_main_to_flib.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:fetch_feed_names.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_attr.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_list_attr.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-control-ret.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-order.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args-handle-info.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-if-ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-while-ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:graph-as-function.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:infer_derived_attribute.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:invalid_input.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:legalized_name.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:missing-main.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:noop.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:optional_symbol_ref.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:output-shapes-attr.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example_v2.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:preserve-entry-func-names.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-type-attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-while-loop.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:shape_list_attr.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple_tf_dialect_op.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:stringescape.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:switchn.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-gradient-attr.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-legacy-call.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_add.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_identity_n.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_tpu_embedding_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_list_attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_name.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_output_name.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:while-loop.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/tf_to_hlo_pipeline:sccp-post-shape-inference.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/tpu_bridge_v1:end_to_end.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/api/v0:compile_mlir_util_test PASSED in 9.7s //tensorflow/compiler/mlir/tf2xla/api/v0:compile_tf_graph_test PASSED in 0.4s //tensorflow/compiler/mlir/tf2xla/api/v1:legalize_tf_test PASSED in 26.9s //tensorflow/compiler/mlir/tf2xla/tests:adjust-layout.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tf2xla/tests:convert-mhlo-quant-to-int.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_runtime_pipeline.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_sparsification.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-BatchMatMulV2.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-binary-elementwise.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-collective.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-communication.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-include-tf2xla-fallback.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-no-tf2xla-fallback.mlir.test PASSED in 6.2s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-prefer-tf2xla.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-types.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-with-tf2xla-hlo-importer.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-with-tf2xla.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf.mlir.test PASSED in 11.7s //tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_cpu.mlir.test PASSED in 3.4s //tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_gpu.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization-no-chlo.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/transforms:tf2xla_rewriter_test PASSED in 16.5s //tensorflow/compiler/mlir/tf2xla/transforms:verify_tfxla_legalization_test PASSED in 18.5s //tensorflow/compiler/mlir/tf2xla/transforms:xla_legalize_targets_test PASSED in 1.3s //tensorflow/compiler/mlir/tf2xla/transforms:xla_legalize_tf_test PASSED in 3.3s //tensorflow/compiler/mlir/tfr:graph_decompose_test PASSED in 11.9s //tensorflow/compiler/mlir/tfr:node_expansion_test PASSED in 7.5s //tensorflow/compiler/mlir/tfr:op_reg_gen_test PASSED in 15.1s //tensorflow/compiler/mlir/tfr:tfr_decompose_ctx_test PASSED in 5.7s //tensorflow/compiler/mlir/tfr:tfr_gen_test PASSED in 19.0s //tensorflow/compiler/mlir/tfr/examples/customization:test_ops_test PASSED in 16.8s //tensorflow/compiler/mlir/tfr/examples/mnist:mnist_ops_test PASSED in 21.3s //tensorflow/compiler/mlir/tfr/examples/pad:pad_ops_test PASSED in 20.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_deallocation.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_reuse.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tools/kernel_gen/tests:bufferize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:copy_cleanup.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tools/kernel_gen/tests:embed_tf_framework.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tools/kernel_gen/tests:invalid.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tools/kernel_gen/tests:isinf.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:ops.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:parallel_loops_to_sequential.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:rewrite_tf_framework_assert.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tanh.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf-legalize-to-lmhlo.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_abi_knowledge.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_framework_legalize_to_llvm.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_kernel_gpu_launch_to_llvm.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_to_jit_invocations.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tosa/tests:convert-tfl-uint8.mlir.test PASSED in 2.4s //tensorflow/compiler/mlir/tosa/tests:convert_metadata.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tosa/tests:fuse-bias-tf.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:lower-complex-types.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tosa/tests:lower_global_tensors.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tosa/tests:multi_add.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tosa/tests:retain_call_once_funcs.mlir.test PASSED in 2.1s //tensorflow/compiler/mlir/tosa/tests:strip-quant-types.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:strip_metadata.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:tf-tfl-to-tosa-pipeline.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:tf-to-tosa-pipeline.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-dequantize_softmax.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline-filtered.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline.mlir.test PASSED in 6.1s //tensorflow/compiler/mlir/tosa/tests:verify_fully_converted.mlir.test PASSED in 0.6s //tensorflow/compiler/tests:adadelta_test_cpu PASSED in 14.0s //tensorflow/compiler/tests:adagrad_da_test_cpu PASSED in 10.5s //tensorflow/compiler/tests:adagrad_test_cpu PASSED in 8.7s //tensorflow/compiler/tests:adam_test_cpu PASSED in 14.2s //tensorflow/compiler/tests:add_n_test_cpu PASSED in 7.9s //tensorflow/compiler/tests:argminmax_test_cpu PASSED in 13.5s //tensorflow/compiler/tests:argminmax_test_cpu_mlir_bridge_test PASSED in 15.3s //tensorflow/compiler/tests:bucketize_op_test_cpu PASSED in 7.6s //tensorflow/compiler/tests:bucketize_op_test_cpu_mlir_bridge_test PASSED in 7.5s //tensorflow/compiler/tests:case_test_cpu PASSED in 7.6s //tensorflow/compiler/tests:cast_ops_test_cpu PASSED in 8.0s //tensorflow/compiler/tests:cast_ops_test_cpu_mlir_bridge_test PASSED in 7.7s //tensorflow/compiler/tests:categorical_op_test_cpu PASSED in 10.0s //tensorflow/compiler/tests:categorical_op_test_cpu_mlir_bridge_test PASSED in 11.3s //tensorflow/compiler/tests:cholesky_op_test_cpu PASSED in 17.1s //tensorflow/compiler/tests:cholesky_op_test_cpu_mlir_bridge_test PASSED in 16.2s //tensorflow/compiler/tests:clustering_test_cpu PASSED in 7.3s //tensorflow/compiler/tests:clustering_test_cpu_mlir_bridge_test PASSED in 7.4s //tensorflow/compiler/tests:concat_ops_test_cpu PASSED in 8.6s //tensorflow/compiler/tests:concat_ops_test_cpu_mlir_bridge_test PASSED in 8.7s //tensorflow/compiler/tests:cond_test_cpu PASSED in 8.3s //tensorflow/compiler/tests:const_arg_test_cpu PASSED in 6.9s //tensorflow/compiler/tests:const_test_cpu PASSED in 8.9s //tensorflow/compiler/tests:data_format_ops_test_cpu PASSED in 12.7s //tensorflow/compiler/tests:data_format_ops_test_cpu_mlir_bridge_test PASSED in 11.6s //tensorflow/compiler/tests:dense_layer_test_cpu PASSED in 11.7s //tensorflow/compiler/tests:dynamic_slice_ops_test_cpu PASSED in 11.9s //tensorflow/compiler/tests:dynamic_slice_ops_test_cpu_mlir_bridge_test PASSED in 13.9s //tensorflow/compiler/tests:dynamic_stitch_test_cpu PASSED in 8.7s //tensorflow/compiler/tests:dynamic_stitch_test_cpu_mlir_bridge_test PASSED in 10.5s //tensorflow/compiler/tests:eager_test_cpu PASSED in 15.4s //tensorflow/compiler/tests:einsum_op_test_cpu PASSED in 8.3s //tensorflow/compiler/tests:einsum_op_test_cpu_mlir_bridge_test PASSED in 17.7s //tensorflow/compiler/tests:ensure_shape_op_test_cpu PASSED in 9.1s //tensorflow/compiler/tests:extract_image_patches_op_test_cpu PASSED in 7.9s //tensorflow/compiler/tests:extract_image_patches_op_test_cpu_mlir_bridge_test PASSED in 7.2s //tensorflow/compiler/tests:fake_quant_ops_test_cpu PASSED in 14.5s //tensorflow/compiler/tests:fake_quant_ops_test_cpu_mlir_bridge_test PASSED in 14.7s //tensorflow/compiler/tests:fifo_queue_test_cpu PASSED in 8.0s //tensorflow/compiler/tests:fifo_queue_test_cpu_mlir_bridge_test PASSED in 8.2s //tensorflow/compiler/tests:ftrl_ops_test_cpu PASSED in 9.3s //tensorflow/compiler/tests:ftrl_ops_test_cpu_mlir_bridge_test PASSED in 12.5s //tensorflow/compiler/tests:function_test_cpu PASSED in 7.9s //tensorflow/compiler/tests:function_test_cpu_mlir_bridge_test PASSED in 6.9s //tensorflow/compiler/tests:gather_nd_op_test_cpu PASSED in 8.4s //tensorflow/compiler/tests:gather_nd_op_test_cpu_mlir_bridge_test PASSED in 9.1s //tensorflow/compiler/tests:gather_test_cpu PASSED in 26.4s //tensorflow/compiler/tests:gather_test_cpu_mlir_bridge_test PASSED in 71.7s //tensorflow/compiler/tests:jit_test_cpu PASSED in 39.0s //tensorflow/compiler/tests:listdiff_op_test_cpu PASSED in 10.3s //tensorflow/compiler/tests:listdiff_op_test_cpu_mlir_bridge_test PASSED in 13.4s //tensorflow/compiler/tests:lrn_ops_test_cpu PASSED in 8.2s //tensorflow/compiler/tests:lrn_ops_test_cpu_mlir_bridge_test PASSED in 6.9s //tensorflow/compiler/tests:lstm_test_cpu PASSED in 23.5s //tensorflow/compiler/tests:manip_ops_test_cpu PASSED in 10.1s //tensorflow/compiler/tests:manip_ops_test_cpu_mlir_bridge_test PASSED in 11.6s //tensorflow/compiler/tests:matrix_band_part_test_cpu PASSED in 47.1s //tensorflow/compiler/tests:matrix_band_part_test_cpu_mlir_bridge_test PASSED in 51.7s //tensorflow/compiler/tests:matrix_inverse_op_test_cpu PASSED in 19.0s //tensorflow/compiler/tests:matrix_inverse_op_test_cpu_mlir_bridge_test PASSED in 19.3s //tensorflow/compiler/tests:matrix_solve_op_test_cpu PASSED in 9.5s //tensorflow/compiler/tests:matrix_solve_op_test_cpu_mlir_bridge_test PASSED in 9.7s //tensorflow/compiler/tests:matrix_triangular_solve_op_test_cpu PASSED in 23.1s //tensorflow/compiler/tests:matrix_triangular_solve_op_test_cpu_mlir_bridge_test PASSED in 24.7s //tensorflow/compiler/tests:momentum_test_cpu PASSED in 9.9s //tensorflow/compiler/tests:nary_ops_test_cpu PASSED in 10.9s //tensorflow/compiler/tests:nary_ops_test_cpu_mlir_bridge_test PASSED in 9.3s //tensorflow/compiler/tests:nullary_ops_test_cpu PASSED in 7.8s //tensorflow/compiler/tests:nullary_ops_test_cpu_mlir_bridge_test PASSED in 11.0s //tensorflow/compiler/tests:placeholder_test_cpu PASSED in 6.4s //tensorflow/compiler/tests:placeholder_test_cpu_mlir_bridge_test PASSED in 8.0s //tensorflow/compiler/tests:proximal_adagrad_test_cpu PASSED in 8.5s //tensorflow/compiler/tests:proximal_gradient_descent_test_cpu PASSED in 8.4s //tensorflow/compiler/tests:quantized_ops_test_cpu PASSED in 8.2s //tensorflow/compiler/tests:reduce_window_test_cpu PASSED in 7.2s //tensorflow/compiler/tests:reduce_window_test_cpu_mlir_bridge_test PASSED in 7.4s //tensorflow/compiler/tests:reshape_op_test_cpu PASSED in 6.4s //tensorflow/compiler/tests:reshape_op_test_cpu_mlir_bridge_test PASSED in 8.7s //tensorflow/compiler/tests:reverse_ops_test_cpu PASSED in 11.9s //tensorflow/compiler/tests:reverse_ops_test_cpu_mlir_bridge_test PASSED in 12.2s //tensorflow/compiler/tests:reverse_sequence_op_test_cpu PASSED in 8.1s //tensorflow/compiler/tests:reverse_sequence_op_test_cpu_mlir_bridge_test PASSED in 11.9s //tensorflow/compiler/tests:risc_ops_test_cpu_mlir_bridge_test PASSED in 14.5s //tensorflow/compiler/tests:rmsprop_test_cpu PASSED in 11.8s //tensorflow/compiler/tests:scatter_nd_op_test_cpu PASSED in 21.6s //tensorflow/compiler/tests:scatter_nd_op_test_cpu_mlir_bridge_test PASSED in 23.7s //tensorflow/compiler/tests:searchsorted_op_test_cpu PASSED in 9.7s //tensorflow/compiler/tests:searchsorted_op_test_cpu_mlir_bridge_test PASSED in 9.8s //tensorflow/compiler/tests:segment_reduction_ops_test_cpu PASSED in 23.4s //tensorflow/compiler/tests:segment_reduction_ops_test_cpu_mlir_bridge_test PASSED in 28.7s //tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu PASSED in 17.4s //tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu_mlir_bridge_test PASSED in 17.8s //tensorflow/compiler/tests:slice_ops_test_cpu PASSED in 15.9s //tensorflow/compiler/tests:slice_ops_test_cpu_mlir_bridge_test PASSED in 21.6s //tensorflow/compiler/tests:sparse_to_dense_op_test_cpu PASSED in 9.7s //tensorflow/compiler/tests:sparse_to_dense_op_test_cpu_mlir_bridge_test PASSED in 7.4s //tensorflow/compiler/tests:stack_ops_test_cpu PASSED in 7.5s //tensorflow/compiler/tests:tensor_float_32_test_cpu PASSED in 8.8s //tensorflow/compiler/tests:tensor_float_32_test_cpu_mlir_bridge_test PASSED in 9.6s //tensorflow/compiler/tests:tensor_list_ops_test_cpu PASSED in 12.4s //tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu PASSED in 13.3s //tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu_mlir_bridge_test PASSED in 26.7s //tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu PASSED in 17.9s //tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu_mlir_bridge_test PASSED in 15.3s //tensorflow/compiler/tests:unique_ops_test_cpu PASSED in 6.5s //tensorflow/compiler/tests:variable_ops_test_cpu PASSED in 22.8s //tensorflow/compiler/tests:variable_ops_test_cpu_mlir_bridge_test PASSED in 14.3s //tensorflow/compiler/tests:where_op_test_cpu PASSED in 8.7s //tensorflow/compiler/tests:while_test_cpu PASSED in 12.1s //tensorflow/compiler/tests:xla_call_module_test_cpu PASSED in 9.7s //tensorflow/compiler/tests:xla_custom_call_ops_test_cpu PASSED in 8.2s //tensorflow/compiler/tests:xla_device_gpu_test_cpu PASSED in 7.7s //tensorflow/compiler/tests:xla_device_test_cpu PASSED in 12.3s //tensorflow/compiler/tests:xla_device_test_cpu_mlir_bridge_test PASSED in 16.4s //tensorflow/compiler/tests:xla_ops_test_cpu PASSED in 36.1s //tensorflow/compiler/tests:xla_ops_test_cpu_mlir_bridge_test PASSED in 38.8s //tensorflow/compiler/tests:xla_test_test PASSED in 6.7s //tensorflow/compiler/tf2xla:const_analysis_test PASSED in 8.0s //tensorflow/compiler/tf2xla:cpu_function_runtime_test PASSED in 0.2s //tensorflow/compiler/tf2xla:functionalize_cond_test PASSED in 1.2s //tensorflow/compiler/tf2xla:functionalize_control_flow_test PASSED in 0.8s //tensorflow/compiler/tf2xla:fused_batchnorm_reserve_space_test_cpu PASSED in 30.6s //tensorflow/compiler/tf2xla:graph_compiler_test PASSED in 6.2s //tensorflow/compiler/tf2xla:literal_util_test PASSED in 0.8s //tensorflow/compiler/tf2xla:resource_operation_table_test PASSED in 8.5s //tensorflow/compiler/tf2xla:resource_util_test_cpu PASSED in 1.8s //tensorflow/compiler/tf2xla:sharding_util_test PASSED in 2.0s //tensorflow/compiler/tf2xla:tf2xla_opset_test PASSED in 10.8s //tensorflow/compiler/tf2xla:tf2xla_test PASSED in 25.8s //tensorflow/compiler/tf2xla:tf2xla_util_test PASSED in 0.8s //tensorflow/compiler/tf2xla:xla_compiler_test PASSED in 22.1s //tensorflow/compiler/tf2xla:xla_jit_compiled_cpu_function_test PASSED in 21.1s //tensorflow/compiler/tf2xla:xla_op_registry_test PASSED in 5.6s //tensorflow/compiler/tf2xla/kernels:rng_converter_utils_test PASSED in 1.8s //tensorflow/compiler/xla:array2d_test PASSED in 0.2s //tensorflow/compiler/xla:array3d_test PASSED in 0.2s //tensorflow/compiler/xla:array4d_test PASSED in 0.2s //tensorflow/compiler/xla:array_test PASSED in 0.1s //tensorflow/compiler/xla:bit_cast_test PASSED in 0.2s //tensorflow/compiler/xla:comparison_util_test PASSED in 0.1s //tensorflow/compiler/xla:debug_options_parsers_test PASSED in 0.2s //tensorflow/compiler/xla:index_util_test PASSED in 0.8s //tensorflow/compiler/xla:iterator_util_test PASSED in 0.4s //tensorflow/compiler/xla:layout_test PASSED in 0.2s //tensorflow/compiler/xla:layout_util_test PASSED in 0.2s //tensorflow/compiler/xla:literal_test PASSED in 0.3s //tensorflow/compiler/xla:parse_flags_from_env_test PASSED in 1.0s //tensorflow/compiler/xla:permutation_util_test PASSED in 0.6s //tensorflow/compiler/xla:primitive_util_test PASSED in 0.1s //tensorflow/compiler/xla:refcounting_hash_map_test PASSED in 0.4s //tensorflow/compiler/xla:reference_util_test PASSED in 0.2s //tensorflow/compiler/xla:shape_test PASSED in 0.1s //tensorflow/compiler/xla:shape_tree_test PASSED in 0.2s //tensorflow/compiler/xla:shape_util_test PASSED in 2.1s //tensorflow/compiler/xla:status_macros_test PASSED in 0.2s //tensorflow/compiler/xla:text_literal_reader_test PASSED in 0.3s //tensorflow/compiler/xla:text_literal_writer_test PASSED in 0.2s //tensorflow/compiler/xla:types_test PASSED in 0.2s //tensorflow/compiler/xla:util_test PASSED in 0.1s //tensorflow/compiler/xla:window_util_test PASSED in 1.5s //tensorflow/compiler/xla/client:padding_test PASSED in 0.2s //tensorflow/compiler/xla/client:xla_builder_test PASSED in 0.3s //tensorflow/compiler/xla/client/lib:arithmetic_test_cpu PASSED in 11.8s //tensorflow/compiler/xla/client/lib:comparators_test_cpu PASSED in 11.7s //tensorflow/compiler/xla/client/lib:constants_test_cpu PASSED in 9.9s //tensorflow/compiler/xla/client/lib:logdet_test_cpu PASSED in 12.5s //tensorflow/compiler/xla/client/lib:math_test_cpu PASSED in 19.1s //tensorflow/compiler/xla/client/lib:matrix_test_cpu PASSED in 13.3s //tensorflow/compiler/xla/client/lib:pooling_test_cpu PASSED in 10.4s //tensorflow/compiler/xla/client/lib:qr_test_cpu PASSED in 17.4s //tensorflow/compiler/xla/client/lib:slicing_test_cpu PASSED in 11.4s //tensorflow/compiler/xla/client/lib:sorting_test_cpu PASSED in 12.7s //tensorflow/compiler/xla/examples/axpy:stablehlo_compile_test PASSED in 16.0s //tensorflow/compiler/xla/experiments/sm_bandwidth_benchmark:sm_bw_test PASSED in 0.1s //tensorflow/compiler/xla/hlo/evaluator:hlo_evaluator_test PASSED in 3.6s //tensorflow/compiler/xla/hlo/experimental/auto_sharding:auto_sharding_test PASSED in 7.0s //tensorflow/compiler/xla/hlo/transforms:hlo_constant_splitter_test PASSED in 1.3s //tensorflow/compiler/xla/hlo/utils:hlo_live_range_test PASSED in 1.3s //tensorflow/compiler/xla/hlo/utils:hlo_matchers_test PASSED in 0.9s //tensorflow/compiler/xla/hlo/utils:hlo_sharding_util_test PASSED in 0.2s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:collective_ops.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:fft.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:legalize_i1_vector_transfers.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:library_ops_to_cpu_runtime.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:lmhlo_custom_call.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:remove_copies_to_out_params.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:rng_bit_generator.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_abi_legalization.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_infeed.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_memref_element_cast_to_llvm.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_outfeed.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:add_hlo_trace.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_launch.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memcpy.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memset.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_case.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_custom_call.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_fft.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cholesky.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_conv.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cublas_lt_matmul.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_gemm.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_infeed.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_outfeed.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_send_recv.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_while.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:memref_get_global_to_arg.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:outline_cuda_graphs.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir/framework/tests:legalize-xla-framework.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/framework/tests:outline-with-xla-framework.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/framework/tests:xla-framework.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/math/transforms/tests:math_optimization.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/memref/transforms/tests:aligned_allocations.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/runtime/ir/tests:ops.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/runtime/ir/tests:ops_verify.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/runtime/ir/tests:testlib.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/runtime/transforms:calling_convention_test PASSED in 0.3s //tensorflow/compiler/xla/mlir/runtime/transforms:type_converter_test PASSED in 0.2s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:compilation_pipeline.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_asserts.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_custom_calls.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:export_functions.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:ordinal_assignment.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:rt_to_llvm.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:erase-op-without-results.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:inline-scf-while.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:reduce-scf-forall-bounds.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-constant.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-value.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-operand-with-constant.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:return-operands-of-terminator-operands.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:truncate-function.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:bisect.mlir.test PASSED in 2.8s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:no-bug.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:snapshot.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/tools/mlir_replay/public:execution_trace_utils_test PASSED in 0.3s //tensorflow/compiler/xla/mlir/utils:error_util_test PASSED in 0.2s //tensorflow/compiler/xla/mlir/xla_cpu/tests:bufferize.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/xla_cpu/tests:invalid.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/xla_cpu/tests:ops.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/bufferization/hlo_one_shot_bufferize.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_hlo_broadcasts.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_hlo_no_broadcasts.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_mhlo.mlir.test PASSED in 1.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/sparse_chlo_legalize_to_linalg.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/buffer_reuse.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/convert_deallocation_ops_to_llvm.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocate.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocate_invalid.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_ops.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_simplification.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_to_scf.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/split_alloc_tensors.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/add_debug_info.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/bufferization.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/collapse-shape.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/collect_stats.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/compose_extract_insert_slice.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/batch_matmul.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/conv_2d_nhwc_hwcf.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/dot.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/duplicate_fusions.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fibonacci.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fusion_outlining.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fusion_planning_for_cpu.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/inline_fusion_clusters.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_bcast_map.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_matmul.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_reduce_map.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_reshape_map.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/matmul.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_1d.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_1d_map.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_2d.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_window.mlir.test PASSED in 0.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reverse.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/scatter.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/sort.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/transpose.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/greedy_fusion.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/invalid.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/lower_vectors.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/nested_tiling_softmax.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/ops.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/optimize_linalg_ops.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/rewrite_forall_to_for.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/simplify_dead_copy.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/tile_by_one.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/tiling_softmax.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/vectorize_copy.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/vectorize_for_cpu.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-select-and-scatter.mlir.test PASSED in 0.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-affine.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-gpu.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-parallel-loops.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-tensor-op.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/ops.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo_gpu/lhlo_gpu_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/attrs.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/broadcast_propagation.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/bitcast.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/canonicalize.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/concatenate.mlir.test PASSED in 1.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/convert.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/convolution.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/custom_call.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/folder_limit.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reduce.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reshape.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reverse.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/scatter.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/transpose.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/tuple.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/while.mlir.test PASSED in 5.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/constraint_fusion.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/convert_to_signless.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/expand_hlo_tuples.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/expand_ops_simplifier.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/group_reduction_dimensions.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-collapse-elementwise-map.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-einsum-to-dot-general.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-gather-to-torch-index-select.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-rng-to-linalg.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-shape-ops-to-standard.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-sort.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-arithmetic.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo-only-dynamic.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo-unranked.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-linalg.mlir.test PASSED in 2.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-memref-unranked.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-memref.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-stablehlo-experimental.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-stablehlo.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/inlining.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/invalid.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-control-flow.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-hlo-shape-computations.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-mhlo-to-thlo.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-to-std.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/lower-complex.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/lower-general-dot.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/materialize-broadcasts.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/merge_assuming_ops.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_bytecode_customizations.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_dot.mlir.test PASSED in 1.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_gather.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_reduction.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_scatter.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_flatten_tuple.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_infer_shape_type_methods.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_ops_prettyprint.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_reduce_pretty_print.mlir.test PASSED in 5.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/ops.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/optimize-hlo.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/prepare-for-export.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/reify-result-types.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/restrict_max_rank.mlir.test PASSED in 1.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/shape_legalize_to_hlo.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/shape_reification.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sink-constants-to-control-flow.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_gendot_lower.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_lower.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_rewriting.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_transpose.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/stablehlo-legalize-to-hlo.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/symbolic-shape-optimization.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/unfuse_batch_norm.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_bounds.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_conv_op.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_reduce_op.mlir.test PASSED in 1.1s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_reduce_window_op.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_scatter_op.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_select_and_scatter_op.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_while_op.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/while_prettyprint.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/bufferize.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/canonicalize.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/invalid.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/legalize_sort.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/ops.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/tiling.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:alloc_to_arg.mlir.test PASSED in 1.9s //tensorflow/compiler/xla/mlir_hlo/tests:assuming-structural-propagation.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:buffer_packing.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:bufferize.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:bufferize_one_shot.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:collapse_parallel_loops_to_1d_pass.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:detensorize_scf_ops.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:index_type_llvm_lowering.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:legalize-trigonometric-to-approximation.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:lower_index_cast.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:propagate_static_shapes.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:rank-specialization.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:scalarization.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:shape-component-analysis.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:shape_simplification.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:test_userange.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:tile_loops.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:unbufferize.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:unroll-loops.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tools/mlir_interpreter/framework/tests:interpreter_value_test PASSED in 0.1s //tensorflow/compiler/xla/mlir_hlo/tools/mlir_interpreter/framework/tests:tensor_or_memref_test PASSED in 0.3s //tensorflow/compiler/xla/mlir_hlo/tosa/tests:binary.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tosa/tests:nullary.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tosa/tests:prepare-mhlo.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tosa/tests:ternary.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tosa/tests:unary.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/pjrt:host_callback_test PASSED in 0.2s //tensorflow/compiler/xla/pjrt:lru_cache_test PASSED in 0.2s //tensorflow/compiler/xla/pjrt:pjrt_api_test PASSED in 0.2s //tensorflow/compiler/xla/pjrt:pjrt_client_test_cpu PASSED in 10.5s //tensorflow/compiler/xla/pjrt:pjrt_compiler_test PASSED in 0.4s //tensorflow/compiler/xla/pjrt:pjrt_executable_test PASSED in 0.2s //tensorflow/compiler/xla/pjrt:pjrt_stream_executor_client_test PASSED in 13.2s //tensorflow/compiler/xla/pjrt:semaphore_test PASSED in 0.7s //tensorflow/compiler/xla/pjrt:tf_pjrt_client_test PASSED in 9.2s //tensorflow/compiler/xla/pjrt:tfrt_cpu_pjrt_client_test PASSED in 11.1s //tensorflow/compiler/xla/pjrt:tracked_device_buffer_test PASSED in 10.7s //tensorflow/compiler/xla/pjrt:tracked_tfrt_cpu_device_buffer_test PASSED in 0.2s //tensorflow/compiler/xla/pjrt:transpose_test PASSED in 45.0s //tensorflow/compiler/xla/pjrt/c:pjrt_c_api_cpu_test PASSED in 9.7s //tensorflow/compiler/xla/pjrt/c:pjrt_c_api_helpers_test PASSED in 0.4s //tensorflow/compiler/xla/pjrt/distributed:client_server_test PASSED in 41.6s //tensorflow/compiler/xla/pjrt/distributed:service_test PASSED in 11.0s //tensorflow/compiler/xla/pjrt/gpu:se_gpu_pjrt_client_test PASSED in 2.0s //tensorflow/compiler/xla/python:outfeed_receiver_test_cpu PASSED in 12.4s //tensorflow/compiler/xla/python/ifrt:array_test PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:array_test_no_impl PASSED in 0.4s //tensorflow/compiler/xla/python/ifrt:client_test_no_impl PASSED in 0.6s //tensorflow/compiler/xla/python/ifrt:executable_test_no_impl PASSED in 1.7s //tensorflow/compiler/xla/python/ifrt:future_test PASSED in 0.8s //tensorflow/compiler/xla/python/ifrt:index_domain_test PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:index_test PASSED in 0.8s //tensorflow/compiler/xla/python/ifrt:shape_test PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:sharding_test PASSED in 0.6s //tensorflow/compiler/xla/python/ifrt:tuple_test_no_impl PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_array.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_assemble.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_call.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_call_loaded_executable.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_disassemble.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_loaded_executable.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_reshard.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/python/ifrt/support:sharding_param_to_op_sharding_test PASSED in 0.4s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_array_impl_test_tfrt_cpu PASSED in 17.1s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_client_impl_test_tfrt_cpu PASSED in 9.8s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_executable_impl_test_tfrt_cpu PASSED in 9.9s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_tuple_impl_test_tfrt_cpu PASSED in 9.8s //tensorflow/compiler/xla/python_api:xla_literal_test PASSED in 1.0s //tensorflow/compiler/xla/python_api:xla_shape_test PASSED in 0.8s //tensorflow/compiler/xla/rpc:grpc_client_test PASSED in 2.6s //tensorflow/compiler/xla/runtime:arguments_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:async_runtime_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:custom_call_test PASSED in 1.8s //tensorflow/compiler/xla/runtime:diagnostics_test PASSED in 0.7s //tensorflow/compiler/xla/runtime:executable_test PASSED in 2.6s //tensorflow/compiler/xla/runtime:ffi_test PASSED in 1.6s //tensorflow/compiler/xla/runtime:map_by_type_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:module_test PASSED in 0.3s //tensorflow/compiler/xla/runtime:results_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:state_test PASSED in 0.3s //tensorflow/compiler/xla/runtime:symbolic_shape_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:type_id_test PASSED in 0.2s //tensorflow/compiler/xla/service:algebraic_simplifier_overflow_test_cpu PASSED in 11.8s //tensorflow/compiler/xla/service:algebraic_simplifier_test PASSED in 6.0s //tensorflow/compiler/xla/service:all_gather_broadcast_reorder_test PASSED in 1.5s //tensorflow/compiler/xla/service:all_gather_combiner_test PASSED in 2.3s //tensorflow/compiler/xla/service:all_gather_decomposer_test PASSED in 1.5s //tensorflow/compiler/xla/service:all_reduce_combiner_test PASSED in 1.2s //tensorflow/compiler/xla/service:all_reduce_contiguous_test PASSED in 1.3s //tensorflow/compiler/xla/service:all_reduce_folder_test PASSED in 1.5s //tensorflow/compiler/xla/service:all_reduce_promotion_test PASSED in 1.4s //tensorflow/compiler/xla/service:all_reduce_reassociate_test PASSED in 2.3s //tensorflow/compiler/xla/service:all_reduce_simplifier_test PASSED in 1.3s //tensorflow/compiler/xla/service:ar_crs_combiner_test PASSED in 2.1s //tensorflow/compiler/xla/service:async_collective_creator_test PASSED in 1.6s //tensorflow/compiler/xla/service:async_op_canonicalizer_test PASSED in 1.2s //tensorflow/compiler/xla/service:batch_dot_simplification_test PASSED in 1.2s //tensorflow/compiler/xla/service:batchnorm_expander_test_cpu PASSED in 11.9s //tensorflow/compiler/xla/service:bfloat16_conversion_folding_test PASSED in 2.1s //tensorflow/compiler/xla/service:bfloat16_propagation_test PASSED in 2.3s //tensorflow/compiler/xla/service:bitcast_dtypes_expander_test PASSED in 1.2s //tensorflow/compiler/xla/service:broadcast_canonicalizer_test PASSED in 1.5s //tensorflow/compiler/xla/service:buffer_assignment_test PASSED in 20.5s //tensorflow/compiler/xla/service:call_graph_test PASSED in 1.6s //tensorflow/compiler/xla/service:call_inliner_test PASSED in 2.9s //tensorflow/compiler/xla/service:change_op_data_type_test PASSED in 1.5s //tensorflow/compiler/xla/service:collective_ops_utils_test PASSED in 0.3s //tensorflow/compiler/xla/service:collectives_schedule_linearizer_test PASSED in 1.5s //tensorflow/compiler/xla/service:compilation_environments_test PASSED in 0.2s //tensorflow/compiler/xla/service:conditional_canonicalizer_test PASSED in 1.2s //tensorflow/compiler/xla/service:conditional_code_motion_test PASSED in 3.1s //tensorflow/compiler/xla/service:conditional_simplifier_test PASSED in 1.5s //tensorflow/compiler/xla/service:conditional_to_select_test PASSED in 1.5s //tensorflow/compiler/xla/service:convert_async_collectives_to_sync_test PASSED in 2.0s //tensorflow/compiler/xla/service:convert_mover_test PASSED in 1.8s //tensorflow/compiler/xla/service:convert_operand_folding_test PASSED in 1.5s //tensorflow/compiler/xla/service:convolution_4d_expander_test PASSED in 1.1s //tensorflow/compiler/xla/service:convolution_group_converter_test PASSED in 0.9s //tensorflow/compiler/xla/service:convolution_pred_expander_test PASSED in 1.0s //tensorflow/compiler/xla/service:copy_insertion_test PASSED in 1.5s //tensorflow/compiler/xla/service:custom_call_status_test PASSED in 0.4s //tensorflow/compiler/xla/service:defuser_test PASSED in 1.3s //tensorflow/compiler/xla/service:despecializer_test PASSED in 1.8s //tensorflow/compiler/xla/service:dfs_hlo_visitor_with_default_test PASSED in 1.1s //tensorflow/compiler/xla/service:dot_decomposer_test PASSED in 1.4s //tensorflow/compiler/xla/service:dot_merger_test PASSED in 2.3s //tensorflow/compiler/xla/service:dynamic_dimension_inference_test PASSED in 1.6s //tensorflow/compiler/xla/service:dynamic_dimension_simplifier_test PASSED in 1.7s //tensorflow/compiler/xla/service:dynamic_index_splitter_test PASSED in 1.2s //tensorflow/compiler/xla/service:dynamic_padder_test_cpu PASSED in 16.9s //tensorflow/compiler/xla/service:dynamic_parameter_binding_test PASSED in 1.4s //tensorflow/compiler/xla/service:dynamic_update_slice_test_cpu PASSED in 13.0s //tensorflow/compiler/xla/service:elemental_ir_emitter_test_cpu PASSED in 14.3s //tensorflow/compiler/xla/service:flatten_call_graph_test PASSED in 1.2s //tensorflow/compiler/xla/service:float_normalization_test PASSED in 1.7s //tensorflow/compiler/xla/service:fusion_node_indexing_evaluation_test PASSED in 1.6s //tensorflow/compiler/xla/service:gather_expander_test PASSED in 1.0s //tensorflow/compiler/xla/service:gather_simplifier_test PASSED in 2.2s //tensorflow/compiler/xla/service:heap_simulator_test PASSED in 1.5s //tensorflow/compiler/xla/service:hlo_alias_analysis_test PASSED in 1.4s //tensorflow/compiler/xla/service:hlo_casting_utils_test PASSED in 7.9s //tensorflow/compiler/xla/service:hlo_computation_deduplicator_test PASSED in 1.2s //tensorflow/compiler/xla/service:hlo_computation_test PASSED in 3.8s //tensorflow/compiler/xla/service:hlo_constant_folding_test PASSED in 5.2s //tensorflow/compiler/xla/service:hlo_cost_analysis_test PASSED in 11.4s //tensorflow/compiler/xla/service:hlo_creation_utils_test PASSED in 5.1s //tensorflow/compiler/xla/service:hlo_cse_test PASSED in 14.7s //tensorflow/compiler/xla/service:hlo_dataflow_analysis_test PASSED in 2.0s //tensorflow/compiler/xla/service:hlo_dce_test PASSED in 1.6s //tensorflow/compiler/xla/service:hlo_domain_test PASSED in 1.1s //tensorflow/compiler/xla/service:hlo_element_type_converter_test PASSED in 1.4s //tensorflow/compiler/xla/service:hlo_execution_profile_test PASSED in 12.5s //tensorflow/compiler/xla/service:hlo_graph_dumper_test PASSED in 1.4s //tensorflow/compiler/xla/service:hlo_input_output_alias_config_test PASSED in 1.6s //tensorflow/compiler/xla/service:hlo_instruction_test PASSED in 2.1s //tensorflow/compiler/xla/service:hlo_liveness_analysis_test PASSED in 1.5s //tensorflow/compiler/xla/service:hlo_memory_scheduler_test PASSED in 1.3s //tensorflow/compiler/xla/service:hlo_module_dce_test PASSED in 1.4s //tensorflow/compiler/xla/service:hlo_module_metadata_test PASSED in 0.2s //tensorflow/compiler/xla/service:hlo_module_test PASSED in 1.8s //tensorflow/compiler/xla/service:hlo_opcode_test PASSED in 0.3s //tensorflow/compiler/xla/service:hlo_ordering_test PASSED in 1.6s //tensorflow/compiler/xla/service:hlo_parser_test PASSED in 0.4s //tensorflow/compiler/xla/service:hlo_pass_pipeline_test PASSED in 1.3s //tensorflow/compiler/xla/service:hlo_phi_graph_test PASSED in 0.7s //tensorflow/compiler/xla/service:hlo_proto_util_test PASSED in 1.5s //tensorflow/compiler/xla/service:hlo_reachability_test PASSED in 1.6s //tensorflow/compiler/xla/service:hlo_rematerialization_test PASSED in 1.3s //tensorflow/compiler/xla/service:hlo_rematerialization_test_utils_test PASSED in 1.7s //tensorflow/compiler/xla/service:hlo_replication_analysis_test PASSED in 2.4s //tensorflow/compiler/xla/service:hlo_schedule_test PASSED in 1.1s //tensorflow/compiler/xla/service:hlo_sharding_test PASSED in 1.2s //tensorflow/compiler/xla/service:hlo_value_semantics_analysis_test PASSED in 2.2s //tensorflow/compiler/xla/service:hlo_verifier_test PASSED in 1.5s //tensorflow/compiler/xla/service:indexed_array_analysis_test PASSED in 0.9s //tensorflow/compiler/xla/service:instruction_fusion_test PASSED in 1.3s //tensorflow/compiler/xla/service:latency_hiding_scheduler_test PASSED in 1.5s //tensorflow/compiler/xla/service:layout_assignment_test PASSED in 10.1s //tensorflow/compiler/xla/service:layout_normalization_test PASSED in 2.4s //tensorflow/compiler/xla/service:logistic_expander_test PASSED in 1.0s //tensorflow/compiler/xla/service:loop_schedule_linearizer_test PASSED in 1.4s //tensorflow/compiler/xla/service:map_inliner_test PASSED in 0.9s //tensorflow/compiler/xla/service:mapped_ptr_container_sorter_test PASSED in 0.5s //tensorflow/compiler/xla/service:memory_space_assignment_best_fit_repacker_test PASSED in 0.3s //tensorflow/compiler/xla/service:memory_space_assignment_test PASSED in 3.1s //tensorflow/compiler/xla/service:memory_space_propagation_test PASSED in 1.3s //tensorflow/compiler/xla/service:name_uniquer_test PASSED in 0.2s //tensorflow/compiler/xla/service:operand_upcaster_test PASSED in 1.3s //tensorflow/compiler/xla/service:optimize_input_output_buffer_alias_test PASSED in 1.2s //tensorflow/compiler/xla/service:pattern_matcher_gmock_test PASSED in 0.2s //tensorflow/compiler/xla/service:pattern_matcher_test PASSED in 1.7s //tensorflow/compiler/xla/service:profile_guided_latency_estimator_test PASSED in 0.9s //tensorflow/compiler/xla/service:real_imag_expander_test PASSED in 1.4s //tensorflow/compiler/xla/service:reduce_decomposer_test PASSED in 1.6s //tensorflow/compiler/xla/service:reduce_scatter_combiner_test PASSED in 1.1s //tensorflow/compiler/xla/service:reduce_scatter_decomposer_test PASSED in 1.4s //tensorflow/compiler/xla/service:reduce_scatter_reassociate_test PASSED in 1.3s //tensorflow/compiler/xla/service:reshape_decomposer_test PASSED in 1.1s //tensorflow/compiler/xla/service:reshape_mover_test PASSED in 1.1s //tensorflow/compiler/xla/service:result_caster_test PASSED in 1.8s //tensorflow/compiler/xla/service:root_instruction_sinker_test PASSED in 1.1s //tensorflow/compiler/xla/service:scatter_expander_test PASSED in 0.9s //tensorflow/compiler/xla/service:scatter_simplifier_test PASSED in 1.3s //tensorflow/compiler/xla/service:select_and_scatter_expander_test PASSED in 1.3s //tensorflow/compiler/xla/service:shape_inference_test PASSED in 0.4s //tensorflow/compiler/xla/service:shaped_buffer_test PASSED in 11.7s //tensorflow/compiler/xla/service:sharding_propagation_test PASSED in 6.3s //tensorflow/compiler/xla/service:sharding_remover_test PASSED in 1.5s //tensorflow/compiler/xla/service:simplify_fp_conversions_test PASSED in 1.1s //tensorflow/compiler/xla/service:slice_sinker_test PASSED in 1.3s //tensorflow/compiler/xla/service:sort_simplifier_test PASSED in 1.5s //tensorflow/compiler/xla/service:space_to_batch_converter_test PASSED in 1.1s //tensorflow/compiler/xla/service:stable_sort_expander_test PASSED in 1.7s //tensorflow/compiler/xla/service:stochastic_convert_decomposer_test PASSED in 1.3s //tensorflow/compiler/xla/service:stream_pool_test PASSED in 1.2s //tensorflow/compiler/xla/service:topk_rewriter_test PASSED in 5.4s //tensorflow/compiler/xla/service:transpose_folding_test PASSED in 2.1s //tensorflow/compiler/xla/service:tuple_points_to_analysis_test PASSED in 1.4s //tensorflow/compiler/xla/service:tuple_simplifier_test PASSED in 1.7s //tensorflow/compiler/xla/service:tuple_util_test PASSED in 1.2s //tensorflow/compiler/xla/service:while_loop_all_reduce_code_motion_test PASSED in 2.8s //tensorflow/compiler/xla/service:while_loop_analysis_test PASSED in 1.4s //tensorflow/compiler/xla/service:while_loop_concat_code_motion_test PASSED in 1.4s //tensorflow/compiler/xla/service:while_loop_constant_sinking_test PASSED in 2.0s //tensorflow/compiler/xla/service:while_loop_expensive_invariant_code_motion_test PASSED in 1.9s //tensorflow/compiler/xla/service:while_loop_invariant_code_motion_test PASSED in 1.2s //tensorflow/compiler/xla/service:while_loop_simplifier_test PASSED in 2.2s //tensorflow/compiler/xla/service:while_loop_trip_count_annotator_test PASSED in 1.1s //tensorflow/compiler/xla/service:while_util_test PASSED in 2.2s //tensorflow/compiler/xla/service:xla_aot_compile_stablehlo_cpu_test PASSED in 12.9s //tensorflow/compiler/xla/service:xla_debug_info_manager_test PASSED in 1.0s //tensorflow/compiler/xla/service:zero_sized_hlo_elimination_test PASSED in 1.2s //tensorflow/compiler/xla/service/cpu:conv_canonicalization_test PASSED in 1.6s //tensorflow/compiler/xla/service/cpu:cpu_eigen_tensor_alignment_test PASSED in 2.2s //tensorflow/compiler/xla/service/cpu:cpu_instruction_fusion_test PASSED in 1.8s //tensorflow/compiler/xla/service/cpu:cpu_layout_assignment_test PASSED in 2.8s //tensorflow/compiler/xla/service/cpu:ir_emission_utils_test PASSED in 1.0s //tensorflow/compiler/xla/service/cpu:parallel_task_assignment_test PASSED in 3.2s //tensorflow/compiler/xla/service/cpu:runtime_fft_test PASSED in 0.1s //tensorflow/compiler/xla/service/cpu:shape_partition_test PASSED in 1.2s //tensorflow/compiler/xla/service/cpu:xfeed_manager_test PASSED in 0.7s //tensorflow/compiler/xla/service/cpu/tests:cpu_bytesizeof_test PASSED in 0.6s //tensorflow/compiler/xla/service/cpu/tests:cpu_dyn_shape_test PASSED in 10.5s //tensorflow/compiler/xla/service/cpu/tests:cpu_eigen_dot_operation_test PASSED in 10.7s //tensorflow/compiler/xla/service/cpu/tests:cpu_external_constants_test PASSED in 27.3s //tensorflow/compiler/xla/service/cpu/tests:cpu_fusion_test PASSED in 9.0s //tensorflow/compiler/xla/service/cpu/tests:cpu_infeed_test PASSED in 9.6s //tensorflow/compiler/xla/service/cpu/tests:cpu_intrinsic_test PASSED in 14.9s //tensorflow/compiler/xla/service/cpu/tests:cpu_key_value_sort_test PASSED in 10.1s //tensorflow/compiler/xla/service/cpu/tests:cpu_literal_caching_test PASSED in 9.4s //tensorflow/compiler/xla/service/cpu/tests:cpu_noalias_test PASSED in 9.2s //tensorflow/compiler/xla/service/cpu/tests:cpu_outfeed_test PASSED in 9.9s //tensorflow/compiler/xla/service/cpu/tests:cpu_profiling_test PASSED in 9.6s //tensorflow/compiler/xla/service/cpu/tests:cpu_spmd_compile_test PASSED in 9.3s //tensorflow/compiler/xla/service/cpu/tests:cpu_topk_test PASSED in 17.5s //tensorflow/compiler/xla/service/cpu/tests:cpu_vectorization_test PASSED in 11.7s //tensorflow/compiler/xla/service/cpu/tests:cpu_while_test PASSED in 11.9s //tensorflow/compiler/xla/service/cpu/tests:tree_reduction_rewriter_test PASSED in 11.1s //tensorflow/compiler/xla/service/gpu:alias_passthrough_params_test PASSED in 1.3s //tensorflow/compiler/xla/service/gpu:all_reduce_blueconnect_test PASSED in 1.1s //tensorflow/compiler/xla/service/gpu:backend_configs_test PASSED in 1.9s //tensorflow/compiler/xla/service/gpu:cublas_pad_for_gemms_test PASSED in 2.3s //tensorflow/compiler/xla/service/gpu:cudnn_pad_for_convolutions_test PASSED in 5.1s //tensorflow/compiler/xla/service/gpu:cudnn_simplify_padding_test PASSED in 2.2s //tensorflow/compiler/xla/service/gpu:cudnn_support_utils_test PASSED in 2.0s //tensorflow/compiler/xla/service/gpu:cudnn_vectorize_convolutions_test PASSED in 1.2s //tensorflow/compiler/xla/service/gpu:fusion_merger_test PASSED in 2.6s //tensorflow/compiler/xla/service/gpu:gemm_rewriter_triton_test PASSED in 1.7s //tensorflow/compiler/xla/service/gpu:gpu_async_collective_annotator_test PASSED in 0.9s //tensorflow/compiler/xla/service/gpu:gpu_conv_padding_legalization_test PASSED in 1.8s //tensorflow/compiler/xla/service/gpu:gpu_conv_rewriter_test PASSED in 1.6s //tensorflow/compiler/xla/service/gpu:gpu_convert_async_collectives_to_sync_test PASSED in 1.9s //tensorflow/compiler/xla/service/gpu:gpu_fusible_test PASSED in 1.8s //tensorflow/compiler/xla/service/gpu:gpu_hlo_cost_analysis_test PASSED in 1.9s //tensorflow/compiler/xla/service/gpu:gpu_performance_model_test PASSED in 2.1s //tensorflow/compiler/xla/service/gpu:gpu_sanitize_constant_names_test PASSED in 1.7s //tensorflow/compiler/xla/service/gpu:hlo_algorithm_denylist_test PASSED in 0.1s //tensorflow/compiler/xla/service/gpu:hlo_fusion_stats_test PASSED in 1.5s //tensorflow/compiler/xla/service/gpu:instruction_fusion_test PASSED in 2.7s //tensorflow/compiler/xla/service/gpu:ir_emission_utils_test PASSED in 2.5s //tensorflow/compiler/xla/service/gpu:matmul_utils_test PASSED in 1.7s //tensorflow/compiler/xla/service/gpu:move_copy_to_users_test PASSED in 1.7s //tensorflow/compiler/xla/service/gpu:multi_output_fusion_test PASSED in 2.5s //tensorflow/compiler/xla/service/gpu:non_atomically_upgradeable_rw_lock_test PASSED in 0.3s //tensorflow/compiler/xla/service/gpu:reduction_splitter_test PASSED in 2.1s //tensorflow/compiler/xla/service/gpu:scatter_slice_simplifier_test PASSED in 1.3s //tensorflow/compiler/xla/service/gpu:target_util_test PASSED in 0.7s //tensorflow/compiler/xla/service/gpu:topk_splitter_test PASSED in 38.2s //tensorflow/compiler/xla/service/gpu:variadic_op_splitter_test PASSED in 1.6s //tensorflow/compiler/xla/service/gpu:while_transformer_test PASSED in 2.0s //tensorflow/compiler/xla/service/gpu/llvm_gpu_backend:utils_test PASSED in 0.4s //tensorflow/compiler/xla/service/gpu/tests:gpu_reduce_scatter_creator_test PASSED in 1.3s //tensorflow/compiler/xla/service/gpu/tests:reduction_degenerate_dim_remover_test PASSED in 2.4s //tensorflow/compiler/xla/service/gpu/tests:reduction_dimension_grouper_test PASSED in 1.9s //tensorflow/compiler/xla/service/gpu/tests:tree_reduction_rewriter_test PASSED in 2.3s //tensorflow/compiler/xla/service/graphcycles:graphcycles_test PASSED in 1.1s //tensorflow/compiler/xla/service/graphcycles:ordered_set_test PASSED in 0.2s //tensorflow/compiler/xla/service/llvm_ir:alias_analysis_test PASSED in 10.8s //tensorflow/compiler/xla/service/llvm_ir:ir_array_test PASSED in 0.8s //tensorflow/compiler/xla/service/spmd:canonicalize_all_gather_for_cse_test PASSED in 2.9s //tensorflow/compiler/xla/service/spmd:collective_permute_motion_test PASSED in 1.7s //tensorflow/compiler/xla/service/spmd:partition_assignment_test PASSED in 1.4s //tensorflow/compiler/xla/service/spmd:schedule_aware_collective_ops_cse_test PASSED in 1.6s //tensorflow/compiler/xla/service/spmd:spmd_partitioner_test PASSED in 2.9s //tensorflow/compiler/xla/service/spmd:stateful_rng_spmd_partitioner_test PASSED in 1.3s //tensorflow/compiler/xla/stream_executor:dnn_test PASSED in 0.3s //tensorflow/compiler/xla/stream_executor:stream_test PASSED in 0.2s //tensorflow/compiler/xla/stream_executor/host:host_stream_test PASSED in 1.2s //tensorflow/compiler/xla/tests:all_reduce_test_cpu PASSED in 13.8s //tensorflow/compiler/xla/tests:axpy_simple_test_cpu PASSED in 10.8s //tensorflow/compiler/xla/tests:bad_rng_shape_validation_test_cpu PASSED in 13.4s //tensorflow/compiler/xla/tests:binop_scaling_test_cpu PASSED in 11.3s //tensorflow/compiler/xla/tests:bitcast_convert_test_cpu PASSED in 14.6s //tensorflow/compiler/xla/tests:broadcast_simple_test_cpu PASSED in 13.1s //tensorflow/compiler/xla/tests:broadcast_test_cpu PASSED in 13.5s //tensorflow/compiler/xla/tests:buffer_donation_test_cpu PASSED in 13.4s //tensorflow/compiler/xla/tests:call_test_cpu PASSED in 13.5s //tensorflow/compiler/xla/tests:check_execution_arity_test_cpu PASSED in 10.7s //tensorflow/compiler/xla/tests:cholesky_test_cpu PASSED in 16.8s //tensorflow/compiler/xla/tests:client_test_cpu PASSED in 10.9s //tensorflow/compiler/xla/tests:collective_ops_test_cpu PASSED in 23.6s //tensorflow/compiler/xla/tests:compilation_cache_test_cpu PASSED in 11.1s //tensorflow/compiler/xla/tests:compute_constant_test_cpu PASSED in 11.4s //tensorflow/compiler/xla/tests:concat_test_cpu PASSED in 14.2s //tensorflow/compiler/xla/tests:constant_reduction_function_test_cpu PASSED in 11.6s //tensorflow/compiler/xla/tests:constants_test_cpu PASSED in 11.5s //tensorflow/compiler/xla/tests:convert_test_cpu PASSED in 15.3s //tensorflow/compiler/xla/tests:copy_test_cpu PASSED in 11.8s //tensorflow/compiler/xla/tests:cpu_gpu_fusion_test_cpu PASSED in 16.5s //tensorflow/compiler/xla/tests:custom_call_test_cpu PASSED in 12.2s //tensorflow/compiler/xla/tests:deallocation_test_cpu PASSED in 11.1s //tensorflow/compiler/xla/tests:deconstruct_tuple_test_cpu PASSED in 11.6s //tensorflow/compiler/xla/tests:deep_graph_test_cpu PASSED in 14.5s //tensorflow/compiler/xla/tests:execution_profile_test_cpu PASSED in 10.4s //tensorflow/compiler/xla/tests:fft_test_cpu PASSED in 11.5s //tensorflow/compiler/xla/tests:float8_test_cpu PASSED in 10.7s //tensorflow/compiler/xla/tests:floor_ceil_test_cpu PASSED in 10.8s //tensorflow/compiler/xla/tests:fmax_fmin_test_cpu PASSED in 11.8s //tensorflow/compiler/xla/tests:gather_operation_test_cpu PASSED in 15.3s //tensorflow/compiler/xla/tests:get_dimension_size_test_cpu PASSED in 10.5s //tensorflow/compiler/xla/tests:half_test_cpu PASSED in 11.7s //tensorflow/compiler/xla/tests:hlo_metadata_test PASSED in 9.7s //tensorflow/compiler/xla/tests:literal_test_util_test PASSED in 4.0s //tensorflow/compiler/xla/tests:local_client_allocation_test_cpu PASSED in 12.3s //tensorflow/compiler/xla/tests:local_client_aot_test PASSED in 0.6s //tensorflow/compiler/xla/tests:log_test_cpu PASSED in 9.1s //tensorflow/compiler/xla/tests:map_test_cpu PASSED in 14.5s //tensorflow/compiler/xla/tests:matrix_ops_simple_test_cpu PASSED in 19.0s //tensorflow/compiler/xla/tests:multidimensional_slice_test_cpu PASSED in 11.8s //tensorflow/compiler/xla/tests:multiple_devices_on_host_test PASSED in 14.3s //tensorflow/compiler/xla/tests:multithreaded_compilation_test_cpu PASSED in 12.4s //tensorflow/compiler/xla/tests:outfeed_in_nested_computation_test_cpu PASSED in 11.5s //tensorflow/compiler/xla/tests:pad_test_cpu PASSED in 12.6s //tensorflow/compiler/xla/tests:pred_test_cpu PASSED in 12.3s //tensorflow/compiler/xla/tests:query_inferred_shape_test_cpu PASSED in 9.3s //tensorflow/compiler/xla/tests:reduce_hlo_test_cpu PASSED in 13.6s //tensorflow/compiler/xla/tests:reduce_precision_test_cpu PASSED in 11.7s //tensorflow/compiler/xla/tests:replay_test_cpu PASSED in 11.3s //tensorflow/compiler/xla/tests:reshape_motion_test_cpu PASSED in 12.2s //tensorflow/compiler/xla/tests:reverse_test_cpu PASSED in 14.8s //tensorflow/compiler/xla/tests:round_trip_packed_literal_test_cpu PASSED in 11.9s //tensorflow/compiler/xla/tests:round_trip_transfer_test_cpu PASSED in 11.6s //tensorflow/compiler/xla/tests:sample_text_test_cpu PASSED in 12.6s //tensorflow/compiler/xla/tests:scatter_test_cpu PASSED in 14.1s //tensorflow/compiler/xla/tests:select_test_cpu PASSED in 11.2s //tensorflow/compiler/xla/tests:test_utils_test_cpu PASSED in 11.9s //tensorflow/compiler/xla/tests:token_hlo_test_cpu PASSED in 12.1s //tensorflow/compiler/xla/tests:topk_test_cpu PASSED in 11.7s //tensorflow/compiler/xla/tests:transfer_manager_test_cpu PASSED in 20.3s //tensorflow/compiler/xla/tests:transpose_test_cpu PASSED in 12.4s //tensorflow/compiler/xla/tests:tuple_test_cpu PASSED in 13.6s //tensorflow/compiler/xla/tests:unary_op_test_cpu PASSED in 11.5s //tensorflow/compiler/xla/tests:value_inference_test_cpu PASSED in 11.8s //tensorflow/compiler/xla/tests:vector_ops_reduce_test_cpu PASSED in 13.1s //tensorflow/compiler/xla/tests:vector_ops_simple_test_cpu PASSED in 12.1s //tensorflow/compiler/xla/tests:while_test_cpu PASSED in 13.3s //tensorflow/compiler/xla/tests/fuzz:rand_000000_cpu PASSED in 11.3s //tensorflow/compiler/xla/tests/fuzz:rand_000002_cpu PASSED in 10.2s //tensorflow/compiler/xla/tests/fuzz:rand_000003_cpu PASSED in 12.3s //tensorflow/compiler/xla/tests/fuzz:rand_000005_cpu PASSED in 10.4s //tensorflow/compiler/xla/tests/fuzz:rand_000006_cpu PASSED in 9.4s //tensorflow/compiler/xla/tests/fuzz:rand_000007_cpu PASSED in 8.7s //tensorflow/compiler/xla/tests/fuzz:rand_000008_cpu PASSED in 12.9s //tensorflow/compiler/xla/tests/fuzz:rand_000009_cpu PASSED in 11.4s //tensorflow/compiler/xla/tests/fuzz:rand_000010_cpu PASSED in 12.0s //tensorflow/compiler/xla/tests/fuzz:rand_000011_cpu PASSED in 13.0s //tensorflow/compiler/xla/tests/fuzz:rand_000012_cpu PASSED in 11.9s //tensorflow/compiler/xla/tests/fuzz:rand_000013_cpu PASSED in 11.1s //tensorflow/compiler/xla/tests/fuzz:rand_000014_cpu PASSED in 10.6s //tensorflow/compiler/xla/tests/fuzz:rand_000015_cpu PASSED in 10.9s //tensorflow/compiler/xla/tests/fuzz:rand_000016_cpu PASSED in 11.5s //tensorflow/compiler/xla/tests/fuzz:rand_000017_cpu PASSED in 10.1s //tensorflow/compiler/xla/tests/fuzz:rand_000018_cpu PASSED in 11.7s //tensorflow/compiler/xla/tests/fuzz:rand_000019_cpu PASSED in 10.6s //tensorflow/compiler/xla/tests/fuzz:rand_000020_cpu PASSED in 10.1s //tensorflow/compiler/xla/tests/fuzz:rand_000021_cpu PASSED in 11.2s //tensorflow/compiler/xla/tests/fuzz:rand_000022_cpu PASSED in 13.8s //tensorflow/compiler/xla/tests/fuzz:rand_000023_cpu PASSED in 11.1s //tensorflow/compiler/xla/tests/fuzz:rand_000024_cpu PASSED in 12.3s //tensorflow/compiler/xla/tests/fuzz:rand_000025_cpu PASSED in 10.2s //tensorflow/compiler/xla/tests/fuzz:rand_000026_cpu PASSED in 10.4s //tensorflow/compiler/xla/tests/fuzz:rand_000027_cpu PASSED in 11.2s //tensorflow/compiler/xla/tests/fuzz:rand_000028_cpu PASSED in 10.7s //tensorflow/compiler/xla/tests/fuzz:rand_000029_cpu PASSED in 10.7s //tensorflow/compiler/xla/tests/fuzz:rand_000030_cpu PASSED in 10.1s //tensorflow/compiler/xla/tests/fuzz:rand_000031_cpu PASSED in 10.8s //tensorflow/compiler/xla/tests/fuzz:rand_000032_cpu PASSED in 11.6s //tensorflow/compiler/xla/tests/fuzz:rand_000033_cpu PASSED in 12.3s //tensorflow/compiler/xla/tests/fuzz:rand_000034_cpu PASSED in 12.6s //tensorflow/compiler/xla/tests/fuzz:rand_000035_cpu PASSED in 10.7s //tensorflow/compiler/xla/tests/fuzz:rand_000036_cpu PASSED in 12.0s //tensorflow/compiler/xla/tests/fuzz:rand_000037_cpu PASSED in 10.8s //tensorflow/compiler/xla/tests/fuzz:rand_000038_cpu PASSED in 10.7s //tensorflow/compiler/xla/tests/fuzz:rand_000039_cpu PASSED in 10.7s //tensorflow/compiler/xla/tests/fuzz:rand_000040_cpu PASSED in 12.9s //tensorflow/compiler/xla/tests/fuzz:rand_000041_cpu PASSED in 13.1s //tensorflow/compiler/xla/tests/fuzz:rand_000042_cpu PASSED in 10.8s //tensorflow/compiler/xla/tests/fuzz:rand_000043_cpu PASSED in 10.7s //tensorflow/compiler/xla/tests/fuzz:rand_000044_cpu PASSED in 13.5s //tensorflow/compiler/xla/tests/fuzz:rand_000045_cpu PASSED in 11.2s //tensorflow/compiler/xla/tests/fuzz:rand_000046_cpu PASSED in 10.6s //tensorflow/compiler/xla/tests/fuzz:rand_000047_cpu PASSED in 11.3s //tensorflow/compiler/xla/tests/fuzz:rand_000048_cpu PASSED in 11.1s //tensorflow/compiler/xla/tests/fuzz:rand_000049_cpu PASSED in 11.5s //tensorflow/compiler/xla/tests/fuzz:rand_000050_cpu PASSED in 12.0s //tensorflow/compiler/xla/tests/fuzz:rand_000051_cpu PASSED in 12.3s //tensorflow/compiler/xla/tests/fuzz:rand_000052_cpu PASSED in 11.2s //tensorflow/compiler/xla/tests/fuzz:rand_000053_cpu PASSED in 10.5s //tensorflow/compiler/xla/tests/fuzz:rand_000054_cpu PASSED in 12.4s //tensorflow/compiler/xla/tests/fuzz:rand_000056_cpu PASSED in 10.1s //tensorflow/compiler/xla/tests/fuzz:rand_000057_cpu PASSED in 11.7s //tensorflow/compiler/xla/tests/fuzz:rand_000058_cpu PASSED in 12.3s //tensorflow/compiler/xla/tests/fuzz:rand_000059_cpu PASSED in 13.6s //tensorflow/compiler/xla/tests/fuzz:rand_000061_cpu PASSED in 10.6s //tensorflow/compiler/xla/tests/fuzz:rand_000062_cpu PASSED in 13.1s //tensorflow/compiler/xla/tests/fuzz:rand_000063_cpu PASSED in 10.6s //tensorflow/compiler/xla/tests/fuzz:rand_000064_cpu PASSED in 10.2s //tensorflow/compiler/xla/tests/fuzz:rand_000065_cpu PASSED in 11.3s //tensorflow/compiler/xla/tests/fuzz:rand_000066_cpu PASSED in 11.2s //tensorflow/compiler/xla/tests/fuzz:rand_000068_cpu PASSED in 10.9s //tensorflow/compiler/xla/tests/fuzz:rand_000069_cpu PASSED in 10.0s //tensorflow/compiler/xla/tests/fuzz:rand_000070_cpu PASSED in 11.5s //tensorflow/compiler/xla/tests/fuzz:rand_000071_cpu PASSED in 10.5s //tensorflow/compiler/xla/tests/fuzz:rand_000073_cpu PASSED in 11.6s //tensorflow/compiler/xla/tests/fuzz:rand_000074_cpu PASSED in 10.8s //tensorflow/compiler/xla/tests/fuzz:rand_000075_cpu PASSED in 13.5s //tensorflow/compiler/xla/tests/fuzz:rand_000076_cpu PASSED in 9.2s //tensorflow/compiler/xla/tests/fuzz:rand_000077_cpu PASSED in 12.0s //tensorflow/compiler/xla/tests/fuzz:rand_000078_cpu PASSED in 11.8s //tensorflow/compiler/xla/tests/fuzz:rand_000079_cpu PASSED in 13.6s //tensorflow/compiler/xla/tests/fuzz:rand_000080_cpu PASSED in 15.2s //tensorflow/compiler/xla/tests/fuzz:rand_000081_cpu PASSED in 11.4s //tensorflow/compiler/xla/tests/fuzz:rand_000082_cpu PASSED in 13.0s //tensorflow/compiler/xla/tests/fuzz:rand_000083_cpu PASSED in 11.5s //tensorflow/compiler/xla/tests/fuzz:rand_000084_cpu PASSED in 10.9s //tensorflow/compiler/xla/tests/fuzz:rand_000085_cpu PASSED in 11.3s //tensorflow/compiler/xla/tests/fuzz:rand_000086_cpu PASSED in 11.2s //tensorflow/compiler/xla/tests/fuzz:rand_000087_cpu PASSED in 13.0s //tensorflow/compiler/xla/tests/fuzz:rand_000088_cpu PASSED in 12.1s //tensorflow/compiler/xla/tests/fuzz:rand_000089_cpu PASSED in 12.9s //tensorflow/compiler/xla/tests/fuzz:rand_000090_cpu PASSED in 10.1s //tensorflow/compiler/xla/tests/fuzz:rand_000091_cpu PASSED in 11.8s //tensorflow/compiler/xla/tests/fuzz:rand_000092_cpu PASSED in 13.4s //tensorflow/compiler/xla/tests/fuzz:rand_000093_cpu PASSED in 11.6s //tensorflow/compiler/xla/tests/fuzz:rand_000094_cpu PASSED in 10.4s //tensorflow/compiler/xla/tests/fuzz:rand_000095_cpu PASSED in 12.1s //tensorflow/compiler/xla/tests/fuzz:rand_000096_cpu PASSED in 9.7s //tensorflow/compiler/xla/tests/fuzz:rand_000097_cpu PASSED in 13.7s //tensorflow/compiler/xla/tests/fuzz:rand_000098_cpu PASSED in 11.4s //tensorflow/compiler/xla/tests/fuzz:rand_000099_cpu PASSED in 11.1s //tensorflow/compiler/xla/tools:hlo_control_flow_flattening_test PASSED in 2.4s //tensorflow/compiler/xla/tools:hlo_extractor_test PASSED in 1.4s //tensorflow/compiler/xla/tools:hlo_module_loader_test PASSED in 1.3s //tensorflow/compiler/xla/tools:interactive_graphviz_bin_test PASSED in 0.3s //tensorflow/compiler/xla/tools:run_hlo_module_bin_test PASSED in 1.1s //tensorflow/compiler/xla/tools/hlo_bisect:hlo_bisect_state_test PASSED in 1.1s //tensorflow/compiler/xla/translate/hlo_to_mhlo:hlo_utils_test PASSED in 0.9s //tensorflow/compiler/xla/translate/hlo_to_mhlo:mlir_hlo_builder_test PASSED in 1.0s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:bool_compare.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:case_conditional.hlotxt.test PASSED in 0.8s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:dynamic_param.hlo.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:entry_computation_layout.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:frontend_attributes.hlotxt.test PASSED in 0.4s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fully_connected_reference_model.hlotxt.test PASSED in 0.7s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fusion.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:if_conditional.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import.hlotxt.test PASSED in 0.8s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import_async.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:layouts_and_names.hlotxt.test PASSED in 1.1s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:location.hlotxt.test PASSED in 0.8s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:module_attributes.hlo.test PASSED in 0.4s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:simple.hlo.test PASSED in 0.6s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:spmd_module_sharding.hlo.test PASSED in 0.4s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:types.hlotxt.test PASSED in 1.0s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:while.hlotxt.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo:type_to_shape_test PASSED in 0.8s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:add.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:case.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:dynamic.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export-with-layouts.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export.mlir.test PASSED in 4.0s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_and_check_layouts.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_large_constants.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_replicas.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:frontend_attributes.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:fusion.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:if.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:input_output_aliasing.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:layouts_and_names.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:location_to_op_metadata.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:missing_main.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:module_attributes.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:multiple_return_tuple.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:opaque_elements_attr.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:rng_get_and_update_state.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:sharding.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:simple.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:unsupported_type.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:while.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:hlo_text_to_lhlo_no_opt.hlotxt.test PASSED in 1.8s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:no_opt_ops.hlotxt.test PASSED in 0.8s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:non_identity_layouts.hlotxt.test PASSED in 0.9s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:ops.mlir.test PASSED in 2.9s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:passthrough.mlir.test PASSED in 0.6s //tensorflow/core:__tensorflow_core_lib_core_legacy_lib_core_all_tests PASSED in 17.6s //tensorflow/core:__tensorflow_core_lib_gtl_legacy_lib_gtl_tests PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_cell_reader_test PASSED in 34.7s //tensorflow/core:__tensorflow_core_lib_monitoring_collection_registry_test PASSED in 0.3s //tensorflow/core:__tensorflow_core_lib_monitoring_counter_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_monitoring_gauge_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_monitoring_metric_def_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_percentile_sampler_test PASSED in 0.6s //tensorflow/core:__tensorflow_core_lib_monitoring_sampler_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_monitoring_test_utils_test PASSED in 0.4s //tensorflow/core:__tensorflow_core_lib_strings_legacy_low_level_library_tests PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_wav_wav_io_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_util_mkl_util_test_srcs PASSED in 0.1s //tensorflow/core:__tensorflow_tsl_lib_core_legacy_lib_core_all_tests PASSED in 0.6s //tensorflow/core:lib_strings_ordered_code_test PASSED in 1.7s //tensorflow/core:lib_strings_proto_serialization_test PASSED in 0.1s //tensorflow/core/api_def:api_test PASSED in 4.5s //tensorflow/core/api_def:update_api_def_test PASSED in 0.3s //tensorflow/core/common_runtime:all_to_all_test_cpu PASSED in 1.1s //tensorflow/core/common_runtime:arg_ret_placement_test PASSED in 1.2s //tensorflow/core/common_runtime:buf_rendezvous_test PASSED in 1.4s //tensorflow/core/common_runtime:collective_executor_mgr_test PASSED in 0.9s //tensorflow/core/common_runtime:collective_param_resolver_local_test PASSED in 4.9s //tensorflow/core/common_runtime:collective_rma_local_test PASSED in 1.2s //tensorflow/core/common_runtime:composite_device_test PASSED in 0.5s //tensorflow/core/common_runtime:cost_measurement_registry_test PASSED in 2.9s //tensorflow/core/common_runtime:cost_util_test PASSED in 0.1s //tensorflow/core/common_runtime:device_mgr_test PASSED in 0.9s //tensorflow/core/common_runtime:device_propagation_test PASSED in 1.0s //tensorflow/core/common_runtime:device_resolver_local_test PASSED in 1.7s //tensorflow/core/common_runtime:device_set_test PASSED in 1.5s //tensorflow/core/common_runtime:direct_session_test_cpu PASSED in 2.6s //tensorflow/core/common_runtime:direct_session_with_debug_test PASSED in 4.6s //tensorflow/core/common_runtime:direct_session_with_tracking_alloc_test PASSED in 0.8s //tensorflow/core/common_runtime:dynamic_device_mgr_test PASSED in 0.7s //tensorflow/core/common_runtime:eval_const_tensor_test PASSED in 0.8s //tensorflow/core/common_runtime:executor_test PASSED in 1.7s //tensorflow/core/common_runtime:function_optimization_registration_test PASSED in 1.1s //tensorflow/core/common_runtime:function_optimization_registry_no_pass_test PASSED in 1.0s //tensorflow/core/common_runtime:function_optimization_registry_pass_failure_test PASSED in 1.3s //tensorflow/core/common_runtime:function_optimization_registry_test PASSED in 1.9s //tensorflow/core/common_runtime:function_threadpool_test PASSED in 1.1s //tensorflow/core/common_runtime:graph_constructor_test PASSED in 2.2s //tensorflow/core/common_runtime:graph_runner_test PASSED in 1.4s //tensorflow/core/common_runtime:hierarchical_tree_broadcaster_test_cpu PASSED in 4.2s //tensorflow/core/common_runtime:inline_function_utils_test PASSED in 0.7s //tensorflow/core/common_runtime:input_colocation_exemption_registry_test PASSED in 0.4s //tensorflow/core/common_runtime:int32_fulltype_test PASSED in 1.3s //tensorflow/core/common_runtime:isolate_placer_inspection_required_ops_pass_test PASSED in 1.2s //tensorflow/core/common_runtime:lower_case_op_test PASSED in 2.3s //tensorflow/core/common_runtime:lower_function_call_test PASSED in 3.1s //tensorflow/core/common_runtime:lower_functional_ops_test PASSED in 3.4s //tensorflow/core/common_runtime:lower_if_op_test PASSED in 2.5s //tensorflow/core/common_runtime:lower_while_op_test PASSED in 1.7s //tensorflow/core/common_runtime:mkl_cpu_allocator_test PASSED in 0.1s //tensorflow/core/common_runtime:mkl_threadpool_device_test PASSED in 0.6s //tensorflow/core/common_runtime:no_op_cost_measurement_test PASSED in 0.1s //tensorflow/core/common_runtime:null_request_cost_accessor_test PASSED in 0.2s //tensorflow/core/common_runtime:optimization_registry_test PASSED in 0.9s //tensorflow/core/common_runtime:optimize_cross_host_control_deps_test PASSED in 6.9s //tensorflow/core/common_runtime:optimize_function_graph_utils_test PASSED in 1.2s //tensorflow/core/common_runtime:partitioning_utils_test PASSED in 0.9s //tensorflow/core/common_runtime:pending_counts_test PASSED in 1.2s //tensorflow/core/common_runtime:permuter_test_cpu PASSED in 4.2s //tensorflow/core/common_runtime:placer_inspection_required_ops_utils_test PASSED in 0.9s //tensorflow/core/common_runtime:placer_test PASSED in 1.1s //tensorflow/core/common_runtime:process_function_library_runtime_test_cpu PASSED in 0.8s //tensorflow/core/common_runtime:process_util_test PASSED in 0.2s //tensorflow/core/common_runtime:quantize_training_test PASSED in 1.5s //tensorflow/core/common_runtime:rendezvous_util_test PASSED in 0.2s //tensorflow/core/common_runtime:replicate_per_replica_nodes_test PASSED in 0.6s //tensorflow/core/common_runtime:request_cost_accessor_registry_test PASSED in 2.1s //tensorflow/core/common_runtime:request_cost_test PASSED in 0.5s //tensorflow/core/common_runtime:ring_gatherer_test_cpu PASSED in 3.5s //tensorflow/core/common_runtime:ring_reducer_test_cpu PASSED in 7.8s //tensorflow/core/common_runtime:scoped_allocator_mgr_test PASSED in 4.6s //tensorflow/core/common_runtime:session_test PASSED in 1.4s //tensorflow/core/common_runtime:shape_refiner_test PASSED in 1.0s //tensorflow/core/common_runtime:single_threaded_executor_test PASSED in 0.7s //tensorflow/core/common_runtime:threadpool_device_test PASSED in 1.2s //tensorflow/core/common_runtime:type_inference_test PASSED in 2.4s //tensorflow/core/common_runtime/eager:attr_builder_test PASSED in 32.3s //tensorflow/core/common_runtime/eager:context_test PASSED in 10.2s //tensorflow/core/common_runtime/eager:custom_device_test PASSED in 11.7s //tensorflow/core/common_runtime/eager:eager_executor_test PASSED in 15.0s //tensorflow/core/common_runtime/eager:eager_op_rewrite_registry_test PASSED in 1.4s //tensorflow/core/common_runtime/eager:eager_operation_test PASSED in 12.2s //tensorflow/core/common_runtime/eager:execute_node_test PASSED in 11.6s //tensorflow/core/common_runtime/eager:execute_test PASSED in 26.2s //tensorflow/core/common_runtime/eager:kernel_and_device_test PASSED in 1.2s //tensorflow/core/common_runtime/eager:mkl_eager_op_rewrite_test PASSED in 13.3s //tensorflow/core/common_runtime/eager:placement_test PASSED in 12.2s //tensorflow/core/common_runtime/eager:placement_utils_test PASSED in 11.3s //tensorflow/core/common_runtime/eager:tensor_handle_data_test PASSED in 12.6s //tensorflow/core/common_runtime/eager:tensor_handle_test PASSED in 14.6s //tensorflow/core/common_runtime/gpu:gpu_device_on_non_gpu_machine_test PASSED in 0.1s //tensorflow/core/common_runtime/next_pluggable_device/c:plugin_c_api_test PASSED in 38.9s //tensorflow/core/config:flags_py_test PASSED in 7.4s //tensorflow/core/config:flags_test PASSED in 0.3s //tensorflow/core/data:compression_utils_test PASSED in 2.4s //tensorflow/core/data:dataset_utils_test PASSED in 0.9s //tensorflow/core/data:hash_utils_test PASSED in 0.7s //tensorflow/core/data:metric_utils_test PASSED in 5.7s //tensorflow/core/data:name_utils_test PASSED in 0.2s //tensorflow/core/data:rewrite_utils_test PASSED in 1.1s //tensorflow/core/data:serialization_utils_test PASSED in 1.0s //tensorflow/core/data:snapshot_utils_test PASSED in 0.7s //tensorflow/core/data:split_utils_test PASSED in 0.9s //tensorflow/core/data:standalone_save_restore_test PASSED in 4.3s //tensorflow/core/data:standalone_test PASSED in 1.3s //tensorflow/core/data:tfdataz_metrics_test PASSED in 2.5s //tensorflow/core/data:unbounded_thread_pool_test PASSED in 0.5s //tensorflow/core/data/service:auto_shard_rewriter_test PASSED in 1.1s //tensorflow/core/data/service:common_test PASSED in 0.1s //tensorflow/core/data/service:credentials_factory_test PASSED in 0.6s //tensorflow/core/data/service:cross_trainer_cache_test PASSED in 2.1s //tensorflow/core/data/service:data_service_test PASSED in 15.8s //tensorflow/core/data/service:data_transfer_test PASSED in 1.6s //tensorflow/core/data/service:dataset_store_test PASSED in 1.0s //tensorflow/core/data/service:dispatcher_client_test PASSED in 4.2s //tensorflow/core/data/service:dispatcher_state_test PASSED in 0.9s //tensorflow/core/data/service:grpc_dispatcher_impl_test PASSED in 2.8s //tensorflow/core/data/service:grpc_util_test PASSED in 0.7s //tensorflow/core/data/service:grpc_worker_impl_test PASSED in 3.3s //tensorflow/core/data/service:journal_test PASSED in 0.7s //tensorflow/core/data/service:logging_utils_test PASSED in 0.3s //tensorflow/core/data/service:task_runner_test PASSED in 4.7s //tensorflow/core/data/service:test_util_test PASSED in 4.2s //tensorflow/core/data/service:url_test PASSED in 0.2s //tensorflow/core/data/service:utils_test PASSED in 1.5s //tensorflow/core/data/service:validate_utils_test PASSED in 0.2s //tensorflow/core/data/service:worker_client_test PASSED in 3.9s //tensorflow/core/data/service:worker_impl_test PASSED in 4.4s //tensorflow/core/data/service/client:data_service_client_test PASSED in 4.9s //tensorflow/core/data/service/client:utils_test PASSED in 4.3s //tensorflow/core/data/service/client:validate_utils_test PASSED in 1.9s //tensorflow/core/data/service/snapshot:distributed_snapshot_test PASSED in 20.0s //tensorflow/core/data/service/snapshot:file_utils_test PASSED in 0.6s //tensorflow/core/data/service/snapshot:path_utils_test PASSED in 0.2s //tensorflow/core/data/service/snapshot:snapshot_manager_test PASSED in 3.5s //tensorflow/core/data/service/snapshot:snapshot_split_provider_test PASSED in 1.0s //tensorflow/core/data/service/snapshot:snapshot_stream_writer_checkpoint_test PASSED in 4.0s //tensorflow/core/data/service/snapshot:snapshot_stream_writer_test PASSED in 4.3s //tensorflow/core/data/service/snapshot:utils_test PASSED in 0.2s //tensorflow/core/debug:debug_graph_utils_test PASSED in 0.7s //tensorflow/core/distributed_runtime:call_options_test PASSED in 0.3s //tensorflow/core/distributed_runtime:cluster_function_library_runtime_test PASSED in 4.2s //tensorflow/core/distributed_runtime:collective_param_resolver_distributed_test PASSED in 1.5s //tensorflow/core/distributed_runtime:collective_rma_distributed_test PASSED in 0.8s //tensorflow/core/distributed_runtime:device_resolver_distributed_test PASSED in 1.0s //tensorflow/core/distributed_runtime:message_wrappers_test PASSED in 0.5s //tensorflow/core/distributed_runtime:partial_run_mgr_test PASSED in 0.6s //tensorflow/core/distributed_runtime:recent_request_ids_test PASSED in 0.1s //tensorflow/core/distributed_runtime:request_id_test PASSED in 0.3s //tensorflow/core/distributed_runtime:rpc_collective_executor_mgr_test PASSED in 2.2s //tensorflow/core/distributed_runtime:server_lib_test PASSED in 0.3s //tensorflow/core/distributed_runtime:session_mgr_test PASSED in 1.2s //tensorflow/core/distributed_runtime:tensor_coding_test PASSED in 0.4s //tensorflow/core/distributed_runtime/coordination:coordination_service_barrier_proxy_test PASSED in 2.2s //tensorflow/core/distributed_runtime/eager:eager_service_impl_test PASSED in 20.4s //tensorflow/core/distributed_runtime/eager:remote_mgr_test PASSED in 11.8s //tensorflow/core/distributed_runtime/integration_test:c_api_coordination_test_cpu PASSED in 52.0s //tensorflow/core/distributed_runtime/integration_test:c_api_multi_client_test_cpu PASSED in 36.6s //tensorflow/core/distributed_runtime/integration_test:c_api_recoverable_jobs_test_cpu PASSED in 52.0s //tensorflow/core/distributed_runtime/integration_test:c_api_session_coordination_test_cpu PASSED in 36.7s //tensorflow/core/distributed_runtime/rpc:grpc_tensor_coding_test PASSED in 4.2s //tensorflow/core/distributed_runtime/rpc:grpc_worker_cache_test PASSED in 1.1s //tensorflow/core/distributed_runtime/rpc/eager:grpc_eager_client_test PASSED in 0.7s //tensorflow/core/example:example_parser_configuration_test PASSED in 1.5s //tensorflow/core/example:feature_util_test PASSED in 0.2s //tensorflow/core/framework:allocator_test PASSED in 3.5s //tensorflow/core/framework:attr_value_util_test PASSED in 1.6s //tensorflow/core/framework:batch_util_test PASSED in 1.2s //tensorflow/core/framework:bfloat16_test PASSED in 0.8s //tensorflow/core/framework:common_shape_fns_test PASSED in 1.1s //tensorflow/core/framework:dataset_test PASSED in 1.6s //tensorflow/core/framework:device_base_test PASSED in 1.1s //tensorflow/core/framework:disable_jit_test PASSED in 2.0s //tensorflow/core/framework:framework_op_gen_lib_test PASSED in 0.8s //tensorflow/core/framework:framework_op_segment_test PASSED in 1.7s //tensorflow/core/framework:framework_resource_var_test PASSED in 0.1s //tensorflow/core/framework:framework_run_handler_test PASSED in 3.9s //tensorflow/core/framework:framework_run_handler_util_test PASSED in 2.3s //tensorflow/core/framework:full_type_inference_util_test PASSED in 0.7s //tensorflow/core/framework:full_type_util_test PASSED in 1.1s //tensorflow/core/framework:function_test PASSED in 1.1s //tensorflow/core/framework:graph_def_util_test PASSED in 1.4s //tensorflow/core/framework:graph_to_functiondef_test PASSED in 1.4s //tensorflow/core/framework:kernel_def_builder_test PASSED in 1.2s //tensorflow/core/framework:kernel_def_util_test PASSED in 1.5s //tensorflow/core/framework:memory_types_test PASSED in 1.7s //tensorflow/core/framework:model_test PASSED in 1.8s //tensorflow/core/framework:node_def_builder_test PASSED in 1.2s //tensorflow/core/framework:node_def_util_test PASSED in 1.0s //tensorflow/core/framework:node_properties_test PASSED in 0.7s //tensorflow/core/framework:op_compatibility_test PASSED in 1.4s //tensorflow/core/framework:op_def_builder_test PASSED in 1.5s //tensorflow/core/framework:op_def_util_test PASSED in 1.3s //tensorflow/core/framework:op_kernel_test PASSED in 0.9s //tensorflow/core/framework:op_registration_test PASSED in 0.7s //tensorflow/core/framework:partial_tensor_shape_test PASSED in 1.5s //tensorflow/core/framework:rendezvous_test PASSED in 3.9s //tensorflow/core/framework:resource_handle_test PASSED in 0.2s //tensorflow/core/framework:resource_mgr_test PASSED in 2.3s //tensorflow/core/framework:resource_op_kernel_test PASSED in 0.8s //tensorflow/core/framework:shape_inference_test PASSED in 2.0s //tensorflow/core/framework:shape_inference_testutil_test PASSED in 1.0s //tensorflow/core/framework:tensor_shape_test PASSED in 6.2s //tensorflow/core/framework:tensor_slice_test PASSED in 1.7s //tensorflow/core/framework:tensor_test PASSED in 29.1s //tensorflow/core/framework:tensor_testutil_test PASSED in 0.9s //tensorflow/core/framework:tensor_util_test PASSED in 0.9s //tensorflow/core/framework:tracking_allocator_test PASSED in 1.2s //tensorflow/core/framework:types_test PASSED in 1.0s //tensorflow/core/framework:variant_op_registry_test PASSED in 17.3s //tensorflow/core/framework:variant_test PASSED in 1.2s //tensorflow/core/framework/registration:registration_test PASSED in 1.5s //tensorflow/core/function/capture:by_ref_capture_test PASSED in 7.9s //tensorflow/core/function/capture:capture_container_test PASSED in 6.9s //tensorflow/core/function/integration_test:side_inputs_manual_api_test PASSED in 13.7s //tensorflow/core/function/integration_test:side_inputs_test PASSED in 14.3s //tensorflow/core/function/polymorphism:function_cache_test PASSED in 7.1s //tensorflow/core/function/polymorphism:function_type_test PASSED in 13.8s //tensorflow/core/function/polymorphism:type_dispatch_test PASSED in 6.6s //tensorflow/core/function/runtime_client:runtime_client_cc_test PASSED in 50.3s //tensorflow/core/function/trace_type:default_types_test PASSED in 6.7s //tensorflow/core/function/trace_type:serialization_test PASSED in 7.1s //tensorflow/core/function/trace_type:trace_type_test PASSED in 8.7s //tensorflow/core/graph:algorithm_test PASSED in 1.4s //tensorflow/core/graph:collective_order_test PASSED in 1.0s //tensorflow/core/graph:control_flow_test PASSED in 1.0s //tensorflow/core/graph:costmodel_test PASSED in 0.8s //tensorflow/core/graph:edgeset_test PASSED in 0.8s //tensorflow/core/graph:graph_def_builder_test PASSED in 0.9s //tensorflow/core/graph:graph_partition_test PASSED in 1.3s //tensorflow/core/graph:graph_test PASSED in 1.3s //tensorflow/core/graph:node_builder_test PASSED in 1.1s //tensorflow/core/graph:optimizer_cse_test PASSED in 1.0s //tensorflow/core/graph:subgraph_test PASSED in 0.9s //tensorflow/core/graph:tensor_id_test PASSED in 1.3s //tensorflow/core/graph:validate_test PASSED in 1.2s //tensorflow/core/graph/regularization:simple_delete_test PASSED in 0.3s //tensorflow/core/graph/regularization:util_test PASSED in 0.2s //tensorflow/core/grappler:graph_topology_view_test PASSED in 0.2s //tensorflow/core/grappler:graph_view_test PASSED in 2.0s //tensorflow/core/grappler:grappler_item_builder_test PASSED in 2.4s //tensorflow/core/grappler:grappler_item_test PASSED in 4.3s //tensorflow/core/grappler:mutable_graph_view_test PASSED in 2.5s //tensorflow/core/grappler:utils_test PASSED in 2.5s //tensorflow/core/grappler/clusters:single_machine_test PASSED in 23.3s //tensorflow/core/grappler/clusters:virtual_cluster_test PASSED in 3.0s //tensorflow/core/grappler/costs:analytical_cost_estimator_test PASSED in 2.1s //tensorflow/core/grappler/costs:cost_estimator_test PASSED in 0.2s //tensorflow/core/grappler/costs:graph_memory_test PASSED in 1.6s //tensorflow/core/grappler/costs:graph_properties_test PASSED in 4.9s //tensorflow/core/grappler/costs:robust_stats_test PASSED in 0.2s //tensorflow/core/grappler/costs:utils_test PASSED in 1.4s //tensorflow/core/grappler/costs:virtual_placer_test PASSED in 0.6s //tensorflow/core/grappler/costs:virtual_scheduler_test PASSED in 1.7s //tensorflow/core/grappler/graph_analyzer:gen_node_test PASSED in 2.1s //tensorflow/core/grappler/graph_analyzer:graph_analyzer_test PASSED in 1.6s //tensorflow/core/grappler/graph_analyzer:hash_tools_test PASSED in 2.2s //tensorflow/core/grappler/graph_analyzer:sig_node_test PASSED in 2.4s //tensorflow/core/grappler/graph_analyzer:subgraph_test PASSED in 2.4s //tensorflow/core/grappler/inputs:utils_test PASSED in 0.3s //tensorflow/core/grappler/optimizers:arithmetic_optimizer_test_cpu PASSED in 4.6s //tensorflow/core/grappler/optimizers:auto_mixed_precision_test_cpu PASSED in 2.7s //tensorflow/core/grappler/optimizers:auto_parallel_test_cpu PASSED in 4.1s //tensorflow/core/grappler/optimizers:common_subgraph_elimination_test_cpu PASSED in 2.6s //tensorflow/core/grappler/optimizers:custom_graph_optimizer_registry_test_cpu PASSED in 4.2s //tensorflow/core/grappler/optimizers:debug_stripper_test_cpu PASSED in 3.5s //tensorflow/core/grappler/optimizers:dependency_optimizer_test_cpu PASSED in 2.4s //tensorflow/core/grappler/optimizers:evaluation_utils_test PASSED in 0.6s //tensorflow/core/grappler/optimizers:function_api_info_test PASSED in 0.1s //tensorflow/core/grappler/optimizers:function_optimizer_test_cpu PASSED in 3.6s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_test_cpu PASSED in 1.7s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_factory_test PASSED in 0.5s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_test_cpu PASSED in 2.0s //tensorflow/core/grappler/optimizers:graph_optimizer_stage_test_cpu PASSED in 2.0s //tensorflow/core/grappler/optimizers:implementation_selector_test PASSED in 2.9s //tensorflow/core/grappler/optimizers:loop_optimizer_test_cpu PASSED in 2.1s //tensorflow/core/grappler/optimizers:memory_optimizer_test_cpu PASSED in 2.7s //tensorflow/core/grappler/optimizers:meta_optimizer_test_cpu PASSED in 9.4s //tensorflow/core/grappler/optimizers:mkl_remapper_test PASSED in 2.1s //tensorflow/core/grappler/optimizers:model_pruner_test_cpu PASSED in 2.3s //tensorflow/core/grappler/optimizers:pin_to_host_optimizer_test_cpu PASSED in 2.5s //tensorflow/core/grappler/optimizers:remapper_test_cpu PASSED in 3.5s //tensorflow/core/grappler/optimizers:scoped_allocator_optimizer_test PASSED in 2.2s //tensorflow/core/grappler/optimizers:shape_optimizer_test_cpu PASSED in 2.7s //tensorflow/core/grappler/optimizers:static_schedule_test_cpu PASSED in 1.6s //tensorflow/core/grappler/optimizers:tfg_optimizer_hook_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:auto_shard_test PASSED in 1.4s //tensorflow/core/grappler/optimizers/data:autotune_buffer_sizes_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:batch_parallelization_test PASSED in 1.2s //tensorflow/core/grappler/optimizers/data:disable_intra_op_parallelism_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:disable_prefetch_legacy_autotune_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:enable_gradient_descent_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:filter_fusion_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:filter_parallelization_test PASSED in 1.2s //tensorflow/core/grappler/optimizers/data:function_utils_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:fusion_utils_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:graph_utils_test PASSED in 0.9s //tensorflow/core/grappler/optimizers/data:inject_prefetch_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:make_deterministic_test PASSED in 1.1s //tensorflow/core/grappler/optimizers/data:make_sloppy_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:map_and_batch_fusion_test PASSED in 1.5s //tensorflow/core/grappler/optimizers/data:map_and_filter_fusion_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:map_fusion_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:map_parallelization_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:noop_elimination_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:parallel_batch_test PASSED in 1.2s //tensorflow/core/grappler/optimizers/data:replicate_on_split_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:shuffle_and_repeat_fusion_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:slack_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:split_utils_test PASSED in 1.3s //tensorflow/core/grappler/optimizers/data:use_private_thread_pool_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/inference:batch_op_rewriter_test PASSED in 0.2s //tensorflow/core/grappler/utils:canonicalizer_test PASSED in 2.0s //tensorflow/core/grappler/utils:colocation_test PASSED in 0.6s //tensorflow/core/grappler/utils:frame_test PASSED in 0.3s //tensorflow/core/grappler/utils:functions_test PASSED in 1.5s //tensorflow/core/grappler/utils:graph_view_internal_test PASSED in 0.8s //tensorflow/core/grappler/utils:graph_view_test PASSED in 3.1s //tensorflow/core/grappler/utils:grappler_test_test PASSED in 7.0s //tensorflow/core/grappler/utils:pattern_utils_test PASSED in 0.9s //tensorflow/core/grappler/utils:scc_test PASSED in 2.7s //tensorflow/core/grappler/utils:symbolic_shapes_test PASSED in 0.1s //tensorflow/core/grappler/utils:topological_sort_test PASSED in 0.7s //tensorflow/core/grappler/utils:tpu_test PASSED in 0.4s //tensorflow/core/grappler/utils:transitive_fanin_test PASSED in 0.6s //tensorflow/core/grappler/utils:traversal_test PASSED in 0.9s //tensorflow/core/grappler/verifiers:structure_verifier_test PASSED in 1.8s //tensorflow/core/ir:interfaces_test PASSED in 0.3s //tensorflow/core/ir:ops_test PASSED in 0.2s //tensorflow/core/ir:shape_inference_utils_test PASSED in 0.3s //tensorflow/core/ir:tf_op_registry_test PASSED in 0.3s //tensorflow/core/ir:tf_op_wrapper_test PASSED in 0.2s //tensorflow/core/ir:utility_test PASSED in 0.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:arg_as_control_ret.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:backedge_segment.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:empty.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:error_during_backedge.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_case_with_attr_inference.pbtxt.test PASSED in 1.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_if_with_attr_inference.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_iterator_get_next_attr_inference.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_underscore_output_shapes.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_while_with_attr_inference.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infeed_dequeue.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_arg_handle_type.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_with_output_shapes.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_arg_name.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_backedge_input_size.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_duplicated_node_name.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_index.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_name.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_attr_key.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_key.pbtxt.test PASSED in 0.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_name.pbtxt.test PASSED in 0.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_op_type.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_func_with_empty_name.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_function_import.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_control_result.pbtxt.test PASSED in 2.2s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_input.pbtxt.test PASSED in 0.4s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_name.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_result.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_attr_name.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_named_edge_index.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_handle_data.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_input.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result_value.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result_value.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_input.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_two_inputs.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_named_edge_index.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_op_name.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_type_list.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:legacy_call.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_shape.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_zero_constant.pbtxt.test PASSED in 1.1s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:three_nodes_with_attrs.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:version.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:empty.mlir.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:fulltype.mlir.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:func_with_no_args_or_results.mlir.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:negative_zero_constant.mlir.test PASSED in 2.0s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:nested_legacy_call.mlir.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:three_nodes_with_attrs.mlir.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:version.mlir.test PASSED in 1.6s //tensorflow/core/ir/importexport/tests/saved_model:saved_model_roundtrip_test PASSED in 0.7s //tensorflow/core/ir/tests:attributes.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:canonicalize.mlir.test PASSED in 1.3s //tensorflow/core/ir/tests:compatible_types.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:concrete-ops.mlir.test PASSED in 0.8s //tensorflow/core/ir/tests:generic_concrete_ops.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:invalid-concrete-ops.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:invalid-preserved-attrs.mlir.test PASSED in 0.4s //tensorflow/core/ir/tests:invalid.mlir.test PASSED in 0.4s //tensorflow/core/ir/tests:invalid_types.mlir.test PASSED in 0.8s //tensorflow/core/ir/tests:ops.mlir.test PASSED in 0.8s //tensorflow/core/ir/tests:region-invalid-ops.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:region-ops-graph.mlir.test PASSED in 0.8s //tensorflow/core/ir/tests:region-ops.mlir.test PASSED in 0.8s //tensorflow/core/ir/tests:types.mlir.test PASSED in 0.6s //tensorflow/core/ir/types:dialect_test PASSED in 0.2s //tensorflow/core/kernels:as_string_op_test PASSED in 0.8s //tensorflow/core/kernels:basic_ops_benchmark_test PASSED in 0.9s //tensorflow/core/kernels:batch_kernels_env_test PASSED in 0.7s //tensorflow/core/kernels:batch_kernels_test PASSED in 1.0s //tensorflow/core/kernels:bias_op_test PASSED in 0.8s //tensorflow/core/kernels:bincount_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:broadcast_to_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:cast_op_test_cpu PASSED in 1.7s //tensorflow/core/kernels:checkpoint_callback_manager_test PASSED in 1.8s //tensorflow/core/kernels:clustering_ops_test PASSED in 1.1s //tensorflow/core/kernels:composite_tensor_variant_test PASSED in 0.8s //tensorflow/core/kernels:concat_op_test PASSED in 0.8s //tensorflow/core/kernels:constant_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:control_flow_ops_test PASSED in 7.4s //tensorflow/core/kernels:conv_grad_filter_ops_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels:conv_grad_input_ops_benchmark_test_cpu PASSED in 1.0s //tensorflow/core/kernels:conv_ops_benchmark_test_cpu PASSED in 1.1s //tensorflow/core/kernels:conv_ops_test_cpu PASSED in 15.1s //tensorflow/core/kernels:count_ops_test PASSED in 1.0s //tensorflow/core/kernels:cross_op_test PASSED in 0.7s //tensorflow/core/kernels:cwise_ops_test_cpu PASSED in 0.6s //tensorflow/core/kernels:debug_ops_test PASSED in 1.4s //tensorflow/core/kernels:decode_wav_op_test PASSED in 2.0s //tensorflow/core/kernels:deep_conv2d_test PASSED in 0.5s //tensorflow/core/kernels:dequantize_op_test PASSED in 0.6s //tensorflow/core/kernels:diag_op_test_cpu PASSED in 1.0s //tensorflow/core/kernels:dynamic_partition_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:dynamic_stitch_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:eigen_activations_test PASSED in 0.6s //tensorflow/core/kernels:eigen_attention_test PASSED in 0.2s //tensorflow/core/kernels:eigen_backward_cuboid_convolutions_test PASSED in 0.6s //tensorflow/core/kernels:eigen_backward_spatial_convolutions_test PASSED in 0.1s //tensorflow/core/kernels:eigen_benchmark_cpu_test PASSED in 0.2s //tensorflow/core/kernels:eigen_mkldnn_contraction_kernel_test PASSED in 0.3s //tensorflow/core/kernels:eigen_pooling_test PASSED in 0.3s //tensorflow/core/kernels:encode_wav_op_test PASSED in 2.5s //tensorflow/core/kernels:fingerprint_op_test PASSED in 0.6s //tensorflow/core/kernels:fused_batch_norm_ex_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:fused_batch_norm_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:gather_nd_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:gather_op_test_cpu PASSED in 1.3s //tensorflow/core/kernels:guarantee_const_op_test PASSED in 0.7s //tensorflow/core/kernels:identity_n_op_test PASSED in 1.1s //tensorflow/core/kernels:identity_op_test PASSED in 0.9s //tensorflow/core/kernels:immutable_constant_op_test PASSED in 1.5s //tensorflow/core/kernels:in_topk_op_test PASSED in 0.6s //tensorflow/core/kernels:isotonic_regression_op_test PASSED in 0.7s //tensorflow/core/kernels:logging_ops_test PASSED in 2.2s //tensorflow/core/kernels:lookup_ops_test PASSED in 0.7s //tensorflow/core/kernels:loss_test PASSED in 0.2s //tensorflow/core/kernels:lrn_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:matmul_op_test_cpu PASSED in 3.5s //tensorflow/core/kernels:merge_v2_checkpoints_op_test PASSED in 0.7s //tensorflow/core/kernels:mfcc_dct_test PASSED in 0.5s //tensorflow/core/kernels:mfcc_mel_filterbank_test PASSED in 0.5s //tensorflow/core/kernels:mfcc_op_test_cpu PASSED in 2.8s //tensorflow/core/kernels:mfcc_test PASSED in 0.2s //tensorflow/core/kernels:multinomial_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:nn_ops_test_cpu PASSED in 1.0s //tensorflow/core/kernels:one_hot_op_test PASSED in 0.8s //tensorflow/core/kernels:ops_testutil_test PASSED in 0.7s //tensorflow/core/kernels:ops_util_test PASSED in 0.1s //tensorflow/core/kernels:parameterized_truncated_normal_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:parse_tensor_test PASSED in 0.7s //tensorflow/core/kernels:quantization_utils_test PASSED in 0.8s //tensorflow/core/kernels:quantize_and_dequantize_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:quantize_down_and_shrink_range_op_test PASSED in 0.7s //tensorflow/core/kernels:quantize_op_test PASSED in 1.8s //tensorflow/core/kernels:quantized_activation_ops_test PASSED in 1.5s //tensorflow/core/kernels:quantized_add_op_test PASSED in 1.4s //tensorflow/core/kernels:quantized_batch_norm_op_test PASSED in 1.1s //tensorflow/core/kernels:quantized_bias_add_op_test PASSED in 0.7s //tensorflow/core/kernels:quantized_concat_op_test PASSED in 1.3s //tensorflow/core/kernels:quantized_conv_ops_test PASSED in 0.5s //tensorflow/core/kernels:quantized_instance_norm_test PASSED in 1.1s //tensorflow/core/kernels:quantized_matmul_op_test PASSED in 1.2s //tensorflow/core/kernels:quantized_mul_op_test PASSED in 1.6s //tensorflow/core/kernels:quantized_pooling_ops_test PASSED in 1.0s //tensorflow/core/kernels:quantized_reshape_op_test PASSED in 0.7s //tensorflow/core/kernels:quantized_resize_bilinear_op_test PASSED in 2.2s //tensorflow/core/kernels:ragged_fill_empty_rows_op_test PASSED in 1.1s //tensorflow/core/kernels:ragged_gather_op_test PASSED in 0.8s //tensorflow/core/kernels:ragged_range_op_test PASSED in 1.1s //tensorflow/core/kernels:ragged_tensor_from_variant_op_test PASSED in 0.9s //tensorflow/core/kernels:ragged_tensor_to_sparse_kernel_test PASSED in 0.6s //tensorflow/core/kernels:ragged_tensor_to_tensor_op_test PASSED in 0.9s //tensorflow/core/kernels:ragged_tensor_to_variant_op_test PASSED in 1.1s //tensorflow/core/kernels:random_binomial_op_test_cpu PASSED in 1.0s //tensorflow/core/kernels:random_index_shuffle_test PASSED in 0.3s //tensorflow/core/kernels:random_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:random_poisson_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:range_sampler_test PASSED in 0.3s //tensorflow/core/kernels:reduction_ops_test_cpu PASSED in 0.7s //tensorflow/core/kernels:regex_replace_op_test PASSED in 1.0s //tensorflow/core/kernels:requantization_range_op_test PASSED in 0.8s //tensorflow/core/kernels:requantize_op_test PASSED in 0.9s //tensorflow/core/kernels:resource_ops_test PASSED in 0.7s //tensorflow/core/kernels:restore_op_test PASSED in 0.7s //tensorflow/core/kernels:restore_v2_op_test PASSED in 0.9s //tensorflow/core/kernels:reverse_op_test PASSED in 0.7s //tensorflow/core/kernels:roll_op_test PASSED in 1.0s //tensorflow/core/kernels:save_op_test PASSED in 0.9s //tensorflow/core/kernels:save_v2_op_test PASSED in 1.1s //tensorflow/core/kernels:scan_ops_test_cpu PASSED in 0.8s //tensorflow/core/kernels:scatter_nd_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:scatter_op_test PASSED in 0.6s //tensorflow/core/kernels:scoped_allocator_ops_test_cpu PASSED in 6.9s //tensorflow/core/kernels:sdca_ops_test PASSED in 1.5s //tensorflow/core/kernels:segment_reduction_ops_test PASSED in 1.0s //tensorflow/core/kernels:sendrecv_ops_test PASSED in 1.0s //tensorflow/core/kernels:sequence_ops_test PASSED in 0.7s //tensorflow/core/kernels:shape_ops_test PASSED in 1.3s //tensorflow/core/kernels:slice_op_test PASSED in 0.5s //tensorflow/core/kernels:spacetobatch_benchmark_test_cpu PASSED in 0.6s //tensorflow/core/kernels:sparse_add_op_test PASSED in 2.3s //tensorflow/core/kernels:sparse_dense_binary_op_shared_test PASSED in 1.1s //tensorflow/core/kernels:sparse_fill_empty_rows_op_test_cpu PASSED in 1.1s //tensorflow/core/kernels:sparse_matmul_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:sparse_reduce_sum_op_test PASSED in 1.0s //tensorflow/core/kernels:sparse_tensor_dense_matmul_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:sparse_to_dense_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:sparse_utils_test PASSED in 0.6s //tensorflow/core/kernels:sparse_xent_op_test_cpu PASSED in 1.0s //tensorflow/core/kernels:spectrogram_op_test_cpu PASSED in 1.9s //tensorflow/core/kernels:spectrogram_test PASSED in 0.2s //tensorflow/core/kernels:split_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:split_v_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:strided_slice_op_test PASSED in 0.7s //tensorflow/core/kernels:string_format_op_test PASSED in 1.0s //tensorflow/core/kernels:string_ngrams_op_test PASSED in 0.7s //tensorflow/core/kernels:string_split_op_test PASSED in 0.7s //tensorflow/core/kernels:substr_op_test PASSED in 0.7s //tensorflow/core/kernels:summary_audio_op_test PASSED in 0.9s //tensorflow/core/kernels:summary_image_op_test PASSED in 1.5s //tensorflow/core/kernels:summary_op_test PASSED in 1.5s //tensorflow/core/kernels:summary_tensor_op_test PASSED in 0.7s //tensorflow/core/kernels:tensor_cord_test PASSED in 0.1s //tensorflow/core/kernels:tensor_flag_utils_test PASSED in 0.4s //tensorflow/core/kernels:tensor_map_test PASSED in 0.1s //tensorflow/core/kernels:training_ops_test PASSED in 0.9s //tensorflow/core/kernels:transpose_util_test PASSED in 0.4s //tensorflow/core/kernels:unary_ops_composition_test_cpu PASSED in 3.0s //tensorflow/core/kernels:unique_op_test PASSED in 0.8s //tensorflow/core/kernels:variable_ops_test PASSED in 1.7s //tensorflow/core/kernels:while_op_test PASSED in 1.5s //tensorflow/core/kernels:xent_op_test_cpu PASSED in 1.0s //tensorflow/core/kernels/batching_util:basic_batch_scheduler_test PASSED in 0.1s //tensorflow/core/kernels/batching_util:batch_input_task_test PASSED in 0.7s //tensorflow/core/kernels/batching_util:batch_resource_base_test PASSED in 0.3s //tensorflow/core/kernels/batching_util:batch_scheduler_test PASSED in 0.2s //tensorflow/core/kernels/batching_util:bounded_executor_test PASSED in 20.5s //tensorflow/core/kernels/batching_util:input_split_metadata_test PASSED in 0.1s //tensorflow/core/kernels/batching_util:periodic_function_test PASSED in 1.7s //tensorflow/core/kernels/batching_util:serial_device_batch_scheduler_test PASSED in 2.0s //tensorflow/core/kernels/batching_util:shared_batch_scheduler_test PASSED in 4.2s //tensorflow/core/kernels/batching_util:threadsafe_status_test PASSED in 0.2s //tensorflow/core/kernels/data:batch_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:cache_dataset_ops_test PASSED in 1.2s //tensorflow/core/kernels/data:concatenate_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data:filter_dataset_op_test PASSED in 1.5s //tensorflow/core/kernels/data:finalize_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:fixed_length_record_dataset_op_test PASSED in 5.1s //tensorflow/core/kernels/data:flat_map_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:get_options_op_test PASSED in 1.2s //tensorflow/core/kernels/data:interleave_dataset_op_test PASSED in 2.6s //tensorflow/core/kernels/data:iterator_ops_test PASSED in 1.2s //tensorflow/core/kernels/data:map_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:map_defun_op_test PASSED in 0.9s //tensorflow/core/kernels/data:optimize_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:options_dataset_op_test PASSED in 0.5s //tensorflow/core/kernels/data:padded_batch_dataset_op_test PASSED in 2.0s //tensorflow/core/kernels/data:parallel_batch_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:parallel_filter_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:parallel_interleave_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:parallel_map_dataset_op_test PASSED in 1.7s //tensorflow/core/kernels/data:prefetch_autotuner_test PASSED in 0.2s //tensorflow/core/kernels/data:prefetch_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:range_dataset_op_test PASSED in 2.1s //tensorflow/core/kernels/data:reduce_dataset_op_test PASSED in 1.5s //tensorflow/core/kernels/data:repeat_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data:rewrite_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:shard_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:shuffle_dataset_op_test PASSED in 1.4s //tensorflow/core/kernels/data:skip_dataset_op_test PASSED in 1.6s //tensorflow/core/kernels/data:sparse_tensor_slice_dataset_op_test PASSED in 2.2s //tensorflow/core/kernels/data:take_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:tensor_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:tensor_slice_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:text_line_dataset_op_test PASSED in 1.9s //tensorflow/core/kernels/data:tf_record_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:window_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:zip_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data/experimental:assert_next_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data/experimental:assert_prev_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data/experimental:auto_shard_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data/experimental:directed_interleave_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data/experimental:list_dataset_op_test PASSED in 1.7s //tensorflow/core/kernels/data/experimental:map_and_batch_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data/experimental:parallel_interleave_dataset_op_test PASSED in 1.4s //tensorflow/core/kernels/data/experimental:random_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data/experimental:sampling_dataset_op_test PASSED in 1.9s //tensorflow/core/kernels/data/experimental:save_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data/experimental:unique_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/image:adjust_contrast_op_benchmark_test_cpu PASSED in 0.8s //tensorflow/core/kernels/image:adjust_contrast_op_test PASSED in 0.9s //tensorflow/core/kernels/image:colorspace_op_test PASSED in 1.0s //tensorflow/core/kernels/image:crop_and_resize_op_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels/image:crop_and_resize_op_test PASSED in 1.0s //tensorflow/core/kernels/image:encode_jpeg_op_test PASSED in 0.6s //tensorflow/core/kernels/image:mirror_pad_op_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels/image:mirror_pad_op_test PASSED in 1.1s //tensorflow/core/kernels/image:non_max_suppression_op_benchmark_test PASSED in 0.9s //tensorflow/core/kernels/image:non_max_suppression_op_test PASSED in 5.4s //tensorflow/core/kernels/image:resize_area_op_test PASSED in 1.4s //tensorflow/core/kernels/image:resize_benchmark_test_cpu PASSED in 1.0s //tensorflow/core/kernels/image:resize_bicubic_op_test PASSED in 4.4s //tensorflow/core/kernels/image:resize_ops_test_cpu PASSED in 3.8s //tensorflow/core/kernels/image:sampling_kernels_test PASSED in 0.5s //tensorflow/core/kernels/image:scale_and_translate_op_test PASSED in 1.8s //tensorflow/core/kernels/linalg:banded_triangular_solve_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels/linalg:matrix_triangular_solve_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels/mkl:mkl_conv_ops_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_dequantize_op_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_fused_batch_norm_op_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_fused_ops_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_matmul_op_benchmark PASSED in 0.7s //tensorflow/core/kernels/mkl:mkl_qmatmul_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_quantize_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_quantized_concat_op_test PASSED in 0.3s //tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_perchannel_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_test PASSED in 0.7s //tensorflow/core/kernels/mkl:mkl_quantized_pooling_ops_test PASSED in 0.3s //tensorflow/core/kernels/mkl:mkl_relu_op_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_requantize_ops_test PASSED in 0.9s //tensorflow/core/kernels/mkl:mkl_swish_op_test PASSED in 0.4s //tensorflow/core/kernels/mkl:onednn_nn_ops_benchmark PASSED in 0.1s //tensorflow/core/kernels/sparse:kernels_test PASSED in 1.2s //tensorflow/core/kernels/uniform_quant_ops:math_utils_test PASSED in 0.2s //tensorflow/core/kernels/uniform_quant_ops:tensor_utils_test PASSED in 0.2s //tensorflow/core/kernels/uniform_quant_ops:uniform_dequantize_op_test PASSED in 0.7s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantize_op_test PASSED in 0.8s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_add_op_test PASSED in 0.8s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_clip_by_value_op_test PASSED in 0.5s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_convolution_ops_test PASSED in 0.7s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_dot_ops_test PASSED in 1.3s //tensorflow/core/kernels/uniform_quant_ops:uniform_requantize_op_test PASSED in 0.9s //tensorflow/core/lib/db:sqlite_test PASSED in 0.1s //tensorflow/core/lib/gif:lib_gif_io_test PASSED in 1.1s //tensorflow/core/lib/jpeg:lib_jpeg_jpeg_mem_unittest PASSED in 0.5s //tensorflow/core/ops:cudnn_rnn_ops_test_cc PASSED in 1.0s //tensorflow/core/ops:ops_array_grad_test PASSED in 1.5s //tensorflow/core/ops:ops_math_grad_test PASSED in 5.1s //tensorflow/core/ops:ops_tests PASSED in 0.8s //tensorflow/core/ops/compat:backwards_compatibility_test PASSED in 0.5s //tensorflow/core/platform:__tensorflow_tsl_platform_profile_utils_cpu_utils_test PASSED in 0.2s //tensorflow/core/platform:enable_tf2_utils_test PASSED in 0.5s //tensorflow/core/platform:env_test PASSED in 2.6s //tensorflow/core/platform:fake_python_env_test PASSED in 0.1s //tensorflow/core/platform:file_system_test PASSED in 0.3s //tensorflow/core/platform:platform_strings_test PASSED in 0.2s //tensorflow/core/platform:ram_file_system_test PASSED in 25.1s //tensorflow/core/platform:resource_loader_test PASSED in 0.2s //tensorflow/core/platform:vmodule_benchmark_test PASSED in 0.3s //tensorflow/core/platform:vmodule_test PASSED in 0.2s //tensorflow/core/profiler/backends/cpu:host_tracer_test PASSED in 0.1s //tensorflow/core/profiler/convert:hlo_proto_to_graph_view_test PASSED in 0.2s //tensorflow/core/profiler/convert:hlo_proto_to_memory_visualization_utils_test PASSED in 0.3s //tensorflow/core/profiler/convert:op_stats_to_pod_stats_test PASSED in 0.1s //tensorflow/core/profiler/convert:op_stats_to_pod_viewer_test PASSED in 0.1s //tensorflow/core/profiler/convert:op_stats_to_tf_stats_test PASSED in 1.2s //tensorflow/core/profiler/convert:repository_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_kernel_stats_db_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_memory_profile_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_op_metrics_db_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_op_stats_test PASSED in 0.5s //tensorflow/core/profiler/convert:xplane_to_step_events_test PASSED in 0.5s //tensorflow/core/profiler/convert:xplane_to_tf_functions_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_tool_names_test PASSED in 0.4s //tensorflow/core/profiler/convert/trace_viewer:trace_viewer_visibility_test PASSED in 0.2s //tensorflow/core/profiler/internal:tfprof_show_test PASSED in 1.1s //tensorflow/core/profiler/internal:tfprof_stats_test PASSED in 1.2s //tensorflow/core/profiler/internal:tfprof_tensor_test PASSED in 1.5s //tensorflow/core/profiler/internal:tfprof_timeline_test PASSED in 1.0s //tensorflow/core/profiler/internal/advisor:tfprof_advisor_test PASSED in 1.6s //tensorflow/core/profiler/lib:profiler_disabled_test PASSED in 0.2s //tensorflow/core/profiler/utils:derived_timeline_test PASSED in 0.2s //tensorflow/core/profiler/utils:kernel_stats_utils_test PASSED in 0.2s //tensorflow/core/profiler/utils:op_metrics_db_utils_test PASSED in 0.2s //tensorflow/core/profiler/utils:step_intersection_test PASSED in 0.3s //tensorflow/core/summary:schema_test PASSED in 0.2s //tensorflow/core/summary:summary_db_writer_test PASSED in 0.2s //tensorflow/core/summary:summary_file_writer_test PASSED in 0.1s //tensorflow/core/tfrt/common:pjrt_cpu_client_registration_test PASSED in 12.2s //tensorflow/core/tfrt/common:pjrt_state_test PASSED in 10.3s //tensorflow/core/tfrt/common:pjrt_util_test PASSED in 9.6s //tensorflow/core/tfrt/fallback:cost_recorder_test PASSED in 0.1s //tensorflow/core/tfrt/fallback:fallback_state_test PASSED in 0.8s //tensorflow/core/transforms:eval_utils_test PASSED in 1.5s //tensorflow/core/transforms:graph_transform_wrapper_test PASSED in 0.5s //tensorflow/core/util:bcast_test PASSED in 1.4s //tensorflow/core/util:command_line_flags_test PASSED in 1.9s //tensorflow/core/util:debug_data_dumper_test PASSED in 1.5s //tensorflow/core/util:debug_events_writer_test PASSED in 0.2s //tensorflow/core/util:dump_graph_test PASSED in 0.7s //tensorflow/core/util:equal_graph_def_test PASSED in 1.5s //tensorflow/core/util:events_writer_test PASSED in 3.4s //tensorflow/core/util:example_proto_fast_parsing_test PASSED in 1.9s //tensorflow/core/util:example_proto_helper_test PASSED in 1.1s //tensorflow/core/util:exec_on_stall_test PASSED in 2.2s //tensorflow/core/util:fake_clock_env_test PASSED in 2.7s //tensorflow/core/util:incremental_barrier_test PASSED in 1.4s //tensorflow/core/util:matmul_bcast_test PASSED in 1.3s //tensorflow/core/util:memmapped_file_system_test PASSED in 2.6s //tensorflow/core/util:overflow_test PASSED in 0.3s //tensorflow/core/util:presized_cuckoo_map_test PASSED in 2.3s //tensorflow/core/util:ragged_to_dense_util_test PASSED in 0.7s //tensorflow/core/util:reffed_status_callback_test PASSED in 1.4s //tensorflow/core/util:reporter_test PASSED in 1.3s //tensorflow/core/util:saved_tensor_slice_util_test PASSED in 0.9s //tensorflow/core/util:semver_test PASSED in 1.2s //tensorflow/core/util:stat_summarizer_test PASSED in 1.0s //tensorflow/core/util:strided_slice_op_test PASSED in 0.9s //tensorflow/core/util:tensor_format_test PASSED in 1.2s //tensorflow/core/util:tensor_slice_reader_test PASSED in 1.8s //tensorflow/core/util:tensor_slice_set_test PASSED in 1.1s //tensorflow/core/util:tensor_slice_util_test PASSED in 1.4s //tensorflow/core/util:tensor_slice_writer_test PASSED in 2.0s //tensorflow/core/util:work_sharder_test PASSED in 1.6s //tensorflow/core/util/ctc:ctc_beam_search_test PASSED in 0.2s //tensorflow/core/util/proto:descriptor_pool_registry_test PASSED in 0.5s //tensorflow/core/util/proto:proto_utils_test PASSED in 0.6s //tensorflow/core/util/quantization:uniform_quant_ops_params_test PASSED in 1.1s //tensorflow/core/util/sparse:sparse_tensor_test PASSED in 0.2s //tensorflow/core/util/tensor_bundle:tensor_bundle_test PASSED in 32.6s //tensorflow/dtensor/mlir:dtensor_location_test PASSED in 0.1s //tensorflow/dtensor/mlir:group_assignment_test PASSED in 0.1s //tensorflow/dtensor/mlir/tests:annotate_global_shape.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:cluster_function_conversion.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:constant_folding.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:designate_resource_handle_mesh.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:device_mesh_cluster_coarsening.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_all_gather.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_all_scatter.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_combine_optimization.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_lowering.mlir.test PASSED in 1.5s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_scatter_optimization.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_sum_optimization.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_alltoall_lowering.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_layout_must_execute.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:dtensor_layout_to_xla_sharding_op.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:dtensor_mixed_precision_reduce.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:dtensor_reduce_scatter_lowering.mlir.test PASSED in 0.4s //tensorflow/dtensor/mlir/tests:dtensor_remove_dtensorlayout.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:dtensor_replace_auxiliary_layout_op.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:dtensor_replace_relayout_with_identity.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding.mlir.test PASSED in 0.4s //tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding_default.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_xla_spmd_integration.mlir.test PASSED in 0.4s //tensorflow/dtensor/mlir/tests:elide_identity_before_copy_to_mesh.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:function_renaming.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:handle_cross_cluster_dependencies.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:handle_sparsetensors.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:layout_propagation_v2.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:lower_send_recv.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:merge_clusters.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:mesh_propagation.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:multi_device_expansion.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:op_to_device_cluster.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:propagate_default_layout.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:propagate_device_id_to_function.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:restore_and_assign.mlir.test PASSED in 1.3s //tensorflow/dtensor/mlir/tests:restore_shape_inference.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:set_default_sharding.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:sparse_expansion.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:spmd_batchparallel.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:spmd_concat.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_conv.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:spmd_einsum.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:spmd_expansion.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:spmd_io_ops.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:spmd_iterator.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_matmul.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_random.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:spmd_save_restore.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_segment_sum.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:spmd_slice.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:spmd_softmax_loss.mlir.test PASSED in 1.6s //tensorflow/dtensor/mlir/tests:spmd_squeeze.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_var_handle.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:tf_dtensor_ops.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:tpu_add_resource_device_attribute.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:tpu_integration.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:undo_merge_const_across_mesh.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:update_tpu_metadata.mlir.test PASSED in 0.6s //tensorflow/dtensor/python/tests:collective_combine_all_reduce_test_cpu PASSED in 16.0s //tensorflow/dtensor/python/tests:collective_test_cpu PASSED in 16.1s //tensorflow/dtensor/python/tests:config_test_cpu PASSED in 7.1s //tensorflow/dtensor/python/tests:device_test_cpu PASSED in 36.4s //tensorflow/dtensor/python/tests:layout_test_cpu PASSED in 15.6s //tensorflow/dtensor/python/tests:multi_client_test_cpu PASSED in 13.4s //tensorflow/dtensor/python/tests:numpy_util_test_cpu PASSED in 9.4s //tensorflow/dtensor/tests:executable_manager_test PASSED in 35.5s //tensorflow/dtensor/tests:layout_to_xla_sharding_test PASSED in 0.2s //tensorflow/dtensor/tests:spmd_expander_test PASSED in 6.7s //tensorflow/dtensor/tests:tensor_layout_test PASSED in 0.2s //tensorflow/examples/adding_an_op:fact_test PASSED in 14.9s //tensorflow/examples/adding_an_op:zero_out_1_test PASSED in 15.7s //tensorflow/examples/adding_an_op:zero_out_2_test PASSED in 34.3s //tensorflow/examples/adding_an_op:zero_out_3_test PASSED in 14.3s //tensorflow/examples/custom_ops_doc/multiplex_1:multiplex_1_test PASSED in 15.5s //tensorflow/examples/custom_ops_doc/multiplex_2:multiplex_2_test_cpu PASSED in 19.8s //tensorflow/examples/custom_ops_doc/multiplex_3:multiplex_3_test PASSED in 15.6s //tensorflow/examples/custom_ops_doc/multiplex_4:multiplex_4_test PASSED in 15.7s //tensorflow/examples/custom_ops_doc/simple_hash_table:simple_hash_table_test PASSED in 16.9s //tensorflow/examples/custom_ops_doc/sleep:sleep_test PASSED in 16.0s //tensorflow/examples/speech_commands:accuracy_utils_test PASSED in 2.1s //tensorflow/examples/speech_commands:models_test PASSED in 15.9s //tensorflow/examples/speech_commands:recognize_commands_test PASSED in 2.1s //tensorflow/examples/wav_to_spectrogram:wav_to_spectrogram_test PASSED in 2.1s //tensorflow/js:ts_op_gen_test PASSED in 0.3s //tensorflow/python:array_grad_test_cpu PASSED in 10.7s //tensorflow/python:autograph_ops_test PASSED in 7.5s //tensorflow/python:batch_norm_benchmark_cpu PASSED in 8.1s //tensorflow/python:bincount_ops_test_cpu PASSED in 11.7s //tensorflow/python:bitwise_ops_test_cpu PASSED in 12.3s //tensorflow/python:clip_ops_test PASSED in 9.3s //tensorflow/python:clustering_ops_test PASSED in 22.8s //tensorflow/python:collective_ops_benchmark_cpu PASSED in 7.2s //tensorflow/python:collective_ops_gpu_test_2gpu PASSED in 10.2s //tensorflow/python:collective_ops_gpu_test_cpu PASSED in 8.6s //tensorflow/python:collective_ops_test PASSED in 19.5s //tensorflow/python:collective_ops_xla_test PASSED in 7.9s //tensorflow/python:compiled_collective_ops_gpu_test_2gpu PASSED in 7.7s //tensorflow/python:compiled_collective_ops_gpu_test_cpu PASSED in 8.3s //tensorflow/python:concat_benchmark_cpu PASSED in 7.3s //tensorflow/python:control_flow_ops_benchmark_cpu PASSED in 7.3s //tensorflow/python:control_flow_v2_enable_test PASSED in 7.4s //tensorflow/python:control_flow_v2_toggles_test PASSED in 8.2s //tensorflow/python:dequantize_op_test PASSED in 8.0s //tensorflow/python:embedding_ops_test_cpu PASSED in 8.9s //tensorflow/python:factory_ops_test_cpu PASSED in 8.3s //tensorflow/python:functional_ops_test PASSED in 6.9s //tensorflow/python:gradient_checker_v2_test_cpu PASSED in 26.1s //tensorflow/python:gradients_test_cpu PASSED in 15.7s //tensorflow/python:init_ops_test_cpu PASSED in 7.8s //tensorflow/python:init_ops_v2_test_cpu PASSED in 10.0s //tensorflow/python:math_grad_test_cpu PASSED in 27.7s //tensorflow/python:math_ops_linspace_test_cpu PASSED in 7.6s //tensorflow/python:math_ops_test_cpu PASSED in 25.5s //tensorflow/python:matmul_benchmark_cpu PASSED in 6.5s //tensorflow/python:nn_grad_test_cpu PASSED in 8.5s //tensorflow/python:nn_loss_scaling_utilities_test PASSED in 10.9s //tensorflow/python:nn_test_cpu PASSED in 48.5s //tensorflow/python:nn_xent_test_cpu PASSED in 7.5s //tensorflow/python:op_selector_test PASSED in 7.5s //tensorflow/python:ops/array_ops_shape_test PASSED in 9.1s //tensorflow/python:ops/array_ops_test PASSED in 7.0s //tensorflow/python:quantized_conv_ops_test PASSED in 7.9s //tensorflow/python:quantized_ops_test PASSED in 8.0s //tensorflow/python:raw_ops_test_cpu PASSED in 8.5s //tensorflow/python:rnn_grad_test_cpu PASSED in 9.1s //tensorflow/python:script_ops_test PASSED in 7.5s //tensorflow/python:sort_ops_test PASSED in 8.4s //tensorflow/python:sparse_ops_test PASSED in 15.6s //tensorflow/python:split_benchmark_cpu PASSED in 6.6s //tensorflow/python:tensor_array_ops_test PASSED in 6.7s //tensorflow/python:transpose_benchmark_cpu PASSED in 7.5s //tensorflow/python:variable_spec_test PASSED in 12.6s //tensorflow/python/autograph/converters:asserts_test PASSED in 12.7s //tensorflow/python/autograph/converters:break_statements_test PASSED in 13.0s //tensorflow/python/autograph/converters:call_trees_test PASSED in 19.9s //tensorflow/python/autograph/converters:conditional_expressions_test PASSED in 7.0s //tensorflow/python/autograph/converters:continue_statements_test PASSED in 8.2s //tensorflow/python/autograph/converters:control_flow_test PASSED in 14.5s //tensorflow/python/autograph/converters:directives_test PASSED in 7.2s //tensorflow/python/autograph/converters:functions_test PASSED in 7.0s //tensorflow/python/autograph/converters:list_comprehensions_test PASSED in 7.1s //tensorflow/python/autograph/converters:lists_test PASSED in 9.4s //tensorflow/python/autograph/converters:logical_expressions_test PASSED in 8.0s //tensorflow/python/autograph/converters:return_statements_test PASSED in 7.6s //tensorflow/python/autograph/converters:slices_test PASSED in 7.2s //tensorflow/python/autograph/converters:variables_test PASSED in 7.1s //tensorflow/python/autograph/core:converter_test PASSED in 7.4s //tensorflow/python/autograph/core:function_wrappers_test PASSED in 7.3s //tensorflow/python/autograph/impl:api_test PASSED in 13.2s //tensorflow/python/autograph/impl:conversion_test PASSED in 7.6s //tensorflow/python/autograph/lang:special_functions_test PASSED in 7.2s //tensorflow/python/autograph/operators:conditional_expressions_test PASSED in 6.8s //tensorflow/python/autograph/operators:control_flow_test PASSED in 14.3s //tensorflow/python/autograph/operators:data_structures_test PASSED in 7.3s //tensorflow/python/autograph/operators:exceptions_test PASSED in 6.4s //tensorflow/python/autograph/operators:logical_test PASSED in 11.1s //tensorflow/python/autograph/operators:py_builtins_test PASSED in 16.7s //tensorflow/python/autograph/operators:slices_test PASSED in 7.5s //tensorflow/python/autograph/operators:variables_test PASSED in 6.9s //tensorflow/python/autograph/pyct:anno_test PASSED in 7.1s //tensorflow/python/autograph/pyct:ast_util_test PASSED in 7.1s //tensorflow/python/autograph/pyct:cache_test PASSED in 7.8s //tensorflow/python/autograph/pyct:cfg_test PASSED in 6.5s //tensorflow/python/autograph/pyct:error_utils_test PASSED in 9.7s //tensorflow/python/autograph/pyct:inspect_utils_test PASSED in 8.5s //tensorflow/python/autograph/pyct:loader_test PASSED in 6.7s //tensorflow/python/autograph/pyct:naming_test PASSED in 6.7s //tensorflow/python/autograph/pyct:origin_info_test PASSED in 7.9s //tensorflow/python/autograph/pyct:parser_test PASSED in 7.7s //tensorflow/python/autograph/pyct:pretty_printer_test PASSED in 6.0s //tensorflow/python/autograph/pyct:qual_names_test PASSED in 7.3s //tensorflow/python/autograph/pyct:templates_test PASSED in 10.9s //tensorflow/python/autograph/pyct:transformer_test PASSED in 12.5s //tensorflow/python/autograph/pyct:transpiler_test PASSED in 6.6s //tensorflow/python/autograph/pyct/static_analysis:activity_test PASSED in 6.6s //tensorflow/python/autograph/pyct/static_analysis:liveness_test PASSED in 7.6s //tensorflow/python/autograph/pyct/static_analysis:reaching_definitions_test PASSED in 6.6s //tensorflow/python/autograph/pyct/static_analysis:reaching_fndefs_test PASSED in 6.7s //tensorflow/python/autograph/pyct/static_analysis:type_inference_test PASSED in 7.6s //tensorflow/python/autograph/tests:assertion_test PASSED in 14.5s //tensorflow/python/autograph/tests:basic_ifexp_test PASSED in 17.5s //tensorflow/python/autograph/tests:call_to_builtin_function_test PASSED in 15.7s //tensorflow/python/autograph/tests:call_to_lambda_function_test PASSED in 15.0s //tensorflow/python/autograph/tests:call_to_named_tuple_test PASSED in 15.5s //tensorflow/python/autograph/tests:call_to_numpy_function_test PASSED in 15.7s //tensorflow/python/autograph/tests:call_to_print_function_test PASSED in 16.0s //tensorflow/python/autograph/tests:call_to_tf_api_test PASSED in 15.0s //tensorflow/python/autograph/tests:call_to_user_function_test PASSED in 21.1s //tensorflow/python/autograph/tests:composite_names_in_control_flow_test PASSED in 21.4s //tensorflow/python/autograph/tests:cond_basic_test PASSED in 23.2s //tensorflow/python/autograph/tests:datasets_test PASSED in 18.0s //tensorflow/python/autograph/tests:early_return_test PASSED in 20.6s //tensorflow/python/autograph/tests:ext_slice_test PASSED in 16.7s //tensorflow/python/autograph/tests:generator_test PASSED in 14.9s //tensorflow/python/autograph/tests:logical_expression_test PASSED in 16.9s //tensorflow/python/autograph/tests:loop_basic_test PASSED in 102.7s //tensorflow/python/autograph/tests:loop_control_flow_illegal_cases_test PASSED in 17.7s //tensorflow/python/autograph/tests:loop_created_variables_test PASSED in 18.2s //tensorflow/python/autograph/tests:loop_scoping_test PASSED in 19.2s //tensorflow/python/autograph/tests:loop_with_function_call_test PASSED in 38.6s //tensorflow/python/autograph/tests:loop_with_variable_type_illegal_cases_test PASSED in 20.8s //tensorflow/python/autograph/tests:loop_with_variable_type_test PASSED in 32.6s //tensorflow/python/autograph/tests:nested_control_flow_test PASSED in 38.4s //tensorflow/python/autograph/tests:type_annotations_test PASSED in 15.6s //tensorflow/python/autograph/utils:context_managers_test PASSED in 6.3s //tensorflow/python/autograph/utils:misc_test PASSED in 7.6s //tensorflow/python/autograph/utils:tensor_list_test PASSED in 6.7s //tensorflow/python/autograph/utils:tensors_test PASSED in 6.4s //tensorflow/python/checkpoint:benchmarks_test PASSED in 6.6s //tensorflow/python/checkpoint:checkpoint_management_test_cpu PASSED in 15.1s //tensorflow/python/checkpoint:checkpoint_metrics_test PASSED in 13.9s //tensorflow/python/checkpoint:checkpoint_test PASSED in 23.1s //tensorflow/python/checkpoint:checkpoint_view_test PASSED in 7.6s //tensorflow/python/checkpoint:checkpoint_with_v1_optimizers_test PASSED in 11.4s //tensorflow/python/checkpoint:functional_saver_test_cpu PASSED in 8.6s //tensorflow/python/checkpoint:restore_test PASSED in 8.5s //tensorflow/python/checkpoint:save_util_v1_test PASSED in 8.5s //tensorflow/python/checkpoint:saveable_compat_test PASSED in 8.5s //tensorflow/python/checkpoint:tensor_callable_test PASSED in 8.5s //tensorflow/python/checkpoint:trackable_view_test PASSED in 8.3s //tensorflow/python/client:device_lib_test_cpu PASSED in 7.4s //tensorflow/python/client:events_writer_test PASSED in 7.7s //tensorflow/python/client:session_benchmark_cpu PASSED in 7.4s //tensorflow/python/client:session_list_devices_test PASSED in 8.4s //tensorflow/python/client:session_partial_run_test PASSED in 10.5s //tensorflow/python/client:timeline_test_cpu PASSED in 7.2s //tensorflow/python/client:virtual_gpu_test_cpu PASSED in 8.2s //tensorflow/python/compat:compat_test PASSED in 7.4s //tensorflow/python/compat:disable_v2_behavior_test PASSED in 7.4s //tensorflow/python/compiler/mlir:mlir_test PASSED in 7.1s //tensorflow/python/compiler/tensorrt:trt_convert_test_cpu PASSED in 7.5s //tensorflow/python/compiler/tensorrt/test:batch_matmul_test_cpu PASSED in 7.4s //tensorflow/python/compiler/tensorrt/test:biasadd_matmul_test_cpu PASSED in 7.5s //tensorflow/python/compiler/tensorrt/test:binary_tensor_weight_broadcast_test_cpu PASSED in 7.9s //tensorflow/python/compiler/tensorrt/test:bool_test_cpu PASSED in 8.4s //tensorflow/python/compiler/tensorrt/test:cast_test_cpu PASSED in 8.3s //tensorflow/python/compiler/tensorrt/test:concatenation_test_cpu PASSED in 8.4s //tensorflow/python/compiler/tensorrt/test:const_broadcast_test_cpu PASSED in 7.6s //tensorflow/python/compiler/tensorrt/test:data_dependent_shape_test_cpu PASSED in 8.0s //tensorflow/python/compiler/tensorrt/test:dynamic_input_shapes_test_cpu PASSED in 7.5s //tensorflow/python/compiler/tensorrt/test:identity_output_test_cpu PASSED in 8.6s //tensorflow/python/compiler/tensorrt/test:int32_test_cpu PASSED in 10.6s //tensorflow/python/compiler/tensorrt/test:lru_cache_test_cpu PASSED in 9.5s //tensorflow/python/compiler/tensorrt/test:memory_alignment_test_cpu PASSED in 8.2s //tensorflow/python/compiler/tensorrt/test:multi_connection_neighbor_engine_test_cpu PASSED in 8.1s //tensorflow/python/compiler/tensorrt/test:neighboring_engine_test_cpu PASSED in 8.2s //tensorflow/python/compiler/tensorrt/test:quantization_test_cpu PASSED in 7.8s //tensorflow/python/compiler/tensorrt/test:rank_two_test_cpu PASSED in 7.8s //tensorflow/python/compiler/tensorrt/test:reshape_transpose_test_cpu PASSED in 8.2s //tensorflow/python/compiler/tensorrt/test:topk_test_cpu PASSED in 8.0s //tensorflow/python/compiler/tensorrt/test:trt_engine_op_shape_test_cpu PASSED in 8.3s //tensorflow/python/compiler/tensorrt/test:trt_mode_test_cpu PASSED in 7.5s //tensorflow/python/compiler/tensorrt/test:unary_test_cpu PASSED in 9.0s //tensorflow/python/compiler/tensorrt/test:vgg_block_nchw_test_cpu PASSED in 7.3s //tensorflow/python/compiler/tensorrt/test:vgg_block_test_cpu PASSED in 8.0s //tensorflow/python/compiler/xla:jit_compile_test_cpu PASSED in 7.8s //tensorflow/python/compiler/xla:jit_test_cpu PASSED in 11.8s //tensorflow/python/compiler/xla:xla_test_cpu PASSED in 17.3s //tensorflow/python/compiler/xla/experimental:xla_sharding_test PASSED in 8.8s //tensorflow/python/data/benchmarks:batch_benchmark PASSED in 7.2s //tensorflow/python/data/benchmarks:filter_benchmark PASSED in 7.5s //tensorflow/python/data/benchmarks:from_tensor_slices_benchmark PASSED in 8.7s //tensorflow/python/data/benchmarks:interleave_benchmark PASSED in 7.8s //tensorflow/python/data/benchmarks:list_files_benchmark PASSED in 6.9s //tensorflow/python/data/benchmarks:map_benchmark PASSED in 7.3s //tensorflow/python/data/benchmarks:meta_benchmark PASSED in 7.0s //tensorflow/python/data/benchmarks:prefetch_benchmark PASSED in 7.5s //tensorflow/python/data/benchmarks:range_benchmark PASSED in 6.9s //tensorflow/python/data/experimental/benchmarks:autotune_benchmark PASSED in 8.1s //tensorflow/python/data/experimental/benchmarks:csv_dataset_benchmark PASSED in 7.7s //tensorflow/python/data/experimental/benchmarks:map_and_batch_benchmark PASSED in 6.9s //tensorflow/python/data/experimental/benchmarks:map_defun_benchmark PASSED in 8.2s //tensorflow/python/data/experimental/benchmarks:matching_files_benchmark PASSED in 6.6s //tensorflow/python/data/experimental/benchmarks:optimize_benchmark PASSED in 8.8s //tensorflow/python/data/experimental/benchmarks:parameter_value_benchmark PASSED in 7.6s //tensorflow/python/data/experimental/benchmarks:rejection_resample_benchmark PASSED in 6.8s //tensorflow/python/data/experimental/benchmarks:snapshot_dataset_benchmark PASSED in 7.3s //tensorflow/python/data/experimental/benchmarks:unbatch_benchmark PASSED in 7.7s //tensorflow/python/data/experimental/kernel_tests:assert_cardinality_test PASSED in 27.3s //tensorflow/python/data/experimental/kernel_tests:assert_next_test PASSED in 9.2s //tensorflow/python/data/experimental/kernel_tests:assert_prev_test PASSED in 9.5s //tensorflow/python/data/experimental/kernel_tests:checkpoint_input_pipeline_hook_test PASSED in 17.8s //tensorflow/python/data/experimental/kernel_tests:compression_ops_test PASSED in 12.1s //tensorflow/python/data/experimental/kernel_tests:copy_to_device_test_cpu PASSED in 13.9s //tensorflow/python/data/experimental/kernel_tests:dense_to_sparse_batch_test PASSED in 24.6s //tensorflow/python/data/experimental/kernel_tests:from_list_test PASSED in 29.6s //tensorflow/python/data/experimental/kernel_tests:io_test PASSED in 50.1s //tensorflow/python/data/experimental/kernel_tests:lookup_ops_test PASSED in 8.7s //tensorflow/python/data/experimental/kernel_tests:make_csv_dataset_test PASSED in 22.2s //tensorflow/python/data/experimental/kernel_tests:make_saveable_from_iterator_test PASSED in 7.5s //tensorflow/python/data/experimental/kernel_tests:make_tf_record_dataset_test PASSED in 54.7s //tensorflow/python/data/experimental/kernel_tests:map_defun_op_test PASSED in 12.5s //tensorflow/python/data/experimental/kernel_tests:matching_files_dataset_test PASSED in 18.2s //tensorflow/python/data/experimental/kernel_tests:model_dataset_test PASSED in 8.3s //tensorflow/python/data/experimental/kernel_tests:non_serializable_test PASSED in 8.8s //tensorflow/python/data/experimental/kernel_tests:prefetch_to_device_test_cpu PASSED in 10.6s //tensorflow/python/data/experimental/kernel_tests:prefetch_with_slack_test PASSED in 10.9s //tensorflow/python/data/experimental/kernel_tests:shuffle_and_repeat_test PASSED in 27.1s //tensorflow/python/data/experimental/kernel_tests:sleep_test PASSED in 7.3s //tensorflow/python/data/experimental/kernel_tests:tf_record_writer_test PASSED in 9.8s //tensorflow/python/data/experimental/kernel_tests:variant_test PASSED in 9.6s //tensorflow/python/data/experimental/kernel_tests:wrap_unwrap_test_cpu PASSED in 17.4s //tensorflow/python/data/experimental/kernel_tests/optimization:filter_fusion_test PASSED in 29.4s //tensorflow/python/data/experimental/kernel_tests/optimization:filter_parallelization_test PASSED in 50.8s //tensorflow/python/data/experimental/kernel_tests/optimization:grappler_test_cpu PASSED in 8.0s //tensorflow/python/data/experimental/kernel_tests/optimization:make_deterministic_test PASSED in 28.9s //tensorflow/python/data/experimental/kernel_tests/optimization:map_and_batch_fusion_test PASSED in 9.1s //tensorflow/python/data/experimental/kernel_tests/optimization:map_and_filter_fusion_test PASSED in 22.8s //tensorflow/python/data/experimental/kernel_tests/optimization:map_fusion_test PASSED in 33.0s //tensorflow/python/data/experimental/kernel_tests/optimization:map_parallelization_test PASSED in 14.1s //tensorflow/python/data/experimental/kernel_tests/optimization:noop_elimination_test PASSED in 18.0s //tensorflow/python/data/experimental/kernel_tests/service:multi_device_test PASSED in 51.7s //tensorflow/python/data/experimental/service:server_lib_test PASSED in 9.6s //tensorflow/python/data/kernel_tests:as_numpy_iterator_test PASSED in 16.1s //tensorflow/python/data/kernel_tests:bucket_by_sequence_length_test PASSED in 16.2s //tensorflow/python/data/kernel_tests:cache_test PASSED in 51.3s //tensorflow/python/data/kernel_tests:cardinality_test PASSED in 14.1s //tensorflow/python/data/kernel_tests:checkpoint_test PASSED in 14.9s //tensorflow/python/data/kernel_tests:concatenate_test PASSED in 50.4s //tensorflow/python/data/kernel_tests:counter_test PASSED in 27.4s //tensorflow/python/data/kernel_tests:dataset_spec_test PASSED in 7.6s //tensorflow/python/data/kernel_tests:dataset_test PASSED in 26.6s //tensorflow/python/data/kernel_tests:enumerate_test PASSED in 21.1s //tensorflow/python/data/kernel_tests:from_sparse_tensor_slices_test PASSED in 8.3s //tensorflow/python/data/kernel_tests:from_tensor_slices_test PASSED in 54.2s //tensorflow/python/data/kernel_tests:from_tensors_test PASSED in 19.2s //tensorflow/python/data/kernel_tests:get_single_element_test PASSED in 10.5s //tensorflow/python/data/kernel_tests:ignore_errors_test PASSED in 16.6s //tensorflow/python/data/kernel_tests:io_test PASSED in 53.2s //tensorflow/python/data/kernel_tests:iterator_test_cpu PASSED in 18.0s //tensorflow/python/data/kernel_tests:len_test PASSED in 7.2s //tensorflow/python/data/kernel_tests:list_files_test PASSED in 11.6s //tensorflow/python/data/kernel_tests:optional_test_cpu PASSED in 11.1s //tensorflow/python/data/kernel_tests:options_test PASSED in 9.9s //tensorflow/python/data/kernel_tests:placement_test_cpu PASSED in 8.8s //tensorflow/python/data/kernel_tests:prefetch_test PASSED in 75.8s //tensorflow/python/data/kernel_tests:random_test PASSED in 27.7s //tensorflow/python/data/kernel_tests:range_test PASSED in 35.5s //tensorflow/python/data/kernel_tests:rebatch_test PASSED in 8.1s //tensorflow/python/data/kernel_tests:reduce_test_cpu PASSED in 28.2s //tensorflow/python/data/kernel_tests:scan_test_cpu PASSED in 38.7s //tensorflow/python/data/kernel_tests:sparse_batch_test PASSED in 19.6s //tensorflow/python/data/kernel_tests:unbatch_test PASSED in 69.7s //tensorflow/python/data/util:convert_test PASSED in 8.3s //tensorflow/python/data/util:nest_test PASSED in 7.9s //tensorflow/python/data/util:options_test PASSED in 6.9s //tensorflow/python/data/util:random_seed_test PASSED in 8.3s //tensorflow/python/data/util:sparse_test PASSED in 8.1s //tensorflow/python/data/util:structure_test PASSED in 8.5s //tensorflow/python/data/util:traverse_test PASSED in 11.0s //tensorflow/python/debug/cli:analyzer_cli_test_cpu PASSED in 9.9s //tensorflow/python/debug/cli:cli_config_test PASSED in 7.1s //tensorflow/python/debug/cli:cli_shared_test PASSED in 6.8s //tensorflow/python/debug/cli:command_parser_test PASSED in 7.3s //tensorflow/python/debug/cli:curses_ui_test PASSED in 7.2s //tensorflow/python/debug/cli:debugger_cli_common_test PASSED in 7.2s //tensorflow/python/debug/cli:evaluator_test PASSED in 6.6s //tensorflow/python/debug/cli:profile_analyzer_cli_test PASSED in 7.8s //tensorflow/python/debug/cli:readline_ui_test PASSED in 6.8s //tensorflow/python/debug/cli:tensor_format_test PASSED in 14.0s //tensorflow/python/debug/lib:check_numerics_callback_test_cpu PASSED in 11.4s //tensorflow/python/debug/lib:common_test PASSED in 10.0s //tensorflow/python/debug/lib:debug_data_test PASSED in 7.5s //tensorflow/python/debug/lib:debug_events_monitors_test PASSED in 14.6s //tensorflow/python/debug/lib:debug_events_writer_test PASSED in 10.1s //tensorflow/python/debug/lib:debug_gradients_test_cpu PASSED in 7.7s //tensorflow/python/debug/lib:debug_graph_reconstruction_test_cpu PASSED in 9.8s //tensorflow/python/debug/lib:debug_graphs_test PASSED in 6.6s //tensorflow/python/debug/lib:debug_grappler_test_cpu PASSED in 7.3s //tensorflow/python/debug/lib:debug_utils_test PASSED in 6.7s //tensorflow/python/debug/lib:debug_v2_ops_test_cpu PASSED in 16.8s //tensorflow/python/debug/lib:profiling_test PASSED in 7.4s //tensorflow/python/debug/lib:session_debug_file_test_cpu PASSED in 14.8s //tensorflow/python/debug/lib:session_debug_multi_gpu_test_cpu PASSED in 6.8s //tensorflow/python/debug/lib:source_utils_test PASSED in 9.7s //tensorflow/python/debug/wrappers:disk_usage_test PASSED in 7.5s //tensorflow/python/debug/wrappers:dumping_wrapper_test PASSED in 16.1s //tensorflow/python/debug/wrappers:framework_test PASSED in 7.7s //tensorflow/python/debug/wrappers:local_cli_wrapper_test PASSED in 7.2s //tensorflow/python/distribute:checkpoint_utils_test_2gpu PASSED in 10.0s //tensorflow/python/distribute:checkpoint_utils_test_cpu PASSED in 9.2s //tensorflow/python/distribute:checkpointing_test_2gpu PASSED in 9.6s //tensorflow/python/distribute:checkpointing_test_cpu PASSED in 9.5s //tensorflow/python/distribute:collective_all_reduce_strategy_test_2gpu PASSED in 57.9s //tensorflow/python/distribute:collective_all_reduce_strategy_test_cpu PASSED in 58.2s //tensorflow/python/distribute:collective_all_reduce_strategy_test_xla_2gpu PASSED in 21.5s //tensorflow/python/distribute:collective_util_test PASSED in 7.1s //tensorflow/python/distribute:combinations_test_2gpu PASSED in 19.3s //tensorflow/python/distribute:combinations_test_cpu PASSED in 21.6s //tensorflow/python/distribute:cross_device_utils_test_cpu PASSED in 7.0s //tensorflow/python/distribute:custom_training_loop_gradient_test_2gpu PASSED in 17.2s //tensorflow/python/distribute:custom_training_loop_gradient_test_cpu PASSED in 10.7s //tensorflow/python/distribute:device_util_test_cpu PASSED in 11.8s //tensorflow/python/distribute:distribute_coordinator_test PASSED in 15.8s //tensorflow/python/distribute:distribute_lib_test PASSED in 11.6s //tensorflow/python/distribute:distribute_utils_test_2gpu PASSED in 15.3s //tensorflow/python/distribute:distribute_utils_test_cpu PASSED in 8.3s //tensorflow/python/distribute:input_ops_test_cpu PASSED in 13.1s //tensorflow/python/distribute:metrics_v1_test_2gpu PASSED in 32.2s //tensorflow/python/distribute:metrics_v1_test_cpu PASSED in 32.6s //tensorflow/python/distribute:mirrored_values_test_2gpu PASSED in 7.9s //tensorflow/python/distribute:mirrored_values_test_cpu PASSED in 8.4s //tensorflow/python/distribute:mirrored_variable_test_2gpu PASSED in 18.9s //tensorflow/python/distribute:mirrored_variable_test_cpu PASSED in 19.3s //tensorflow/python/distribute:multi_process_runner_no_init_test PASSED in 8.1s //tensorflow/python/distribute:multi_worker_continuous_run_test_cpu PASSED in 22.0s //tensorflow/python/distribute:multi_worker_util_test PASSED in 7.5s //tensorflow/python/distribute:numpy_dataset_test PASSED in 7.0s //tensorflow/python/distribute:one_device_strategy_test_cpu PASSED in 19.3s //tensorflow/python/distribute:packed_distributed_variable_test PASSED in 6.9s //tensorflow/python/distribute:parameter_server_strategy_test_2gpu PASSED in 26.7s //tensorflow/python/distribute:parameter_server_strategy_test_cpu PASSED in 26.6s //tensorflow/python/distribute:parameter_server_strategy_v2_test_2gpu PASSED in 24.0s //tensorflow/python/distribute:parameter_server_strategy_v2_test_cpu PASSED in 22.7s //tensorflow/python/distribute:per_replica_test_2gpu PASSED in 8.9s //tensorflow/python/distribute:per_replica_test_cpu PASSED in 8.0s //tensorflow/python/distribute:ps_values_test_2gpu PASSED in 9.1s //tensorflow/python/distribute:ps_values_test_cpu PASSED in 8.8s //tensorflow/python/distribute:remote_mirrored_strategy_eager_test_cpu PASSED in 9.0s //tensorflow/python/distribute:sharded_variable_test PASSED in 27.8s //tensorflow/python/distribute:shared_variable_creator_test PASSED in 7.6s //tensorflow/python/distribute:strategy_combinations_test_cpu PASSED in 44.3s //tensorflow/python/distribute:template_mirrored_strategy_test_cpu PASSED in 24.2s //tensorflow/python/distribute:test_util_test_2gpu PASSED in 17.3s //tensorflow/python/distribute:test_util_test_cpu PASSED in 16.3s //tensorflow/python/distribute:tf_function_test_2gpu PASSED in 9.6s //tensorflow/python/distribute:tf_function_test_cpu PASSED in 9.7s //tensorflow/python/distribute:values_v2_test_cpu PASSED in 10.8s //tensorflow/python/distribute:warm_starting_util_test_2gpu PASSED in 12.1s //tensorflow/python/distribute:warm_starting_util_test_cpu PASSED in 9.8s //tensorflow/python/distribute/cluster_resolver:base_cluster_resolver_py_test PASSED in 7.6s //tensorflow/python/distribute/cluster_resolver:gce_cluster_resolver_py_test PASSED in 7.3s //tensorflow/python/distribute/cluster_resolver:kubernetes_cluster_resolver_py_test PASSED in 8.1s //tensorflow/python/distribute/cluster_resolver:sagemaker_cluster_resolver_py_test PASSED in 15.3s //tensorflow/python/distribute/cluster_resolver:slurm_cluster_resolver_py_test PASSED in 21.2s //tensorflow/python/distribute/cluster_resolver:tfconfig_cluster_resolver_py_test PASSED in 7.5s //tensorflow/python/distribute/cluster_resolver/tpu:tpu_cluster_resolver_py_test PASSED in 13.2s //tensorflow/python/distribute/coordinator:metric_utils_test PASSED in 9.9s //tensorflow/python/distribute/coordinator:watchdog_test PASSED in 68.1s //tensorflow/python/distribute/experimental:dtensor_util_test_cpu PASSED in 11.9s //tensorflow/python/distribute/experimental:mirrored_strategy_test_cpu PASSED in 30.0s //tensorflow/python/distribute/experimental:multi_worker_mirrored_strategy_test_cpu PASSED in 12.2s //tensorflow/python/distribute/integration_test:saved_model_test_cpu PASSED in 34.9s //tensorflow/python/distribute/parallel_device:parallel_device_test_cpu PASSED in 10.6s //tensorflow/python/distribute/v1:all_reduce_test PASSED in 38.3s //tensorflow/python/distribute/v1:cross_device_ops_test_2gpu PASSED in 46.4s //tensorflow/python/distribute/v1:cross_device_ops_test_cpu PASSED in 60.1s //tensorflow/python/dlpack:dlpack_test_cpu PASSED in 8.6s //tensorflow/python/eager:backprop_test_cpu PASSED in 109.7s //tensorflow/python/eager:benchmarks_test_cpu PASSED in 7.8s //tensorflow/python/eager:cancellation_test_cpu PASSED in 7.0s //tensorflow/python/eager:context_test_cpu PASSED in 8.0s //tensorflow/python/eager:core_test_cpu PASSED in 27.1s //tensorflow/python/eager:gradient_input_output_exclusions_test PASSED in 29.2s //tensorflow/python/eager:graph_only_ops_test_cpu PASSED in 7.1s //tensorflow/python/eager:lift_to_graph_test PASSED in 17.8s //tensorflow/python/eager:monitoring_test_cpu PASSED in 11.8s //tensorflow/python/eager:ops_test_cpu PASSED in 14.7s //tensorflow/python/eager:profiler_client_test PASSED in 7.1s //tensorflow/python/eager:profiler_test_cpu PASSED in 21.4s //tensorflow/python/eager:pywrap_tfe_test PASSED in 18.7s //tensorflow/python/eager:record_test PASSED in 8.1s //tensorflow/python/eager:remote_benchmarks_test_cpu PASSED in 8.2s //tensorflow/python/eager:run_eager_op_as_function_test_cpu PASSED in 7.9s //tensorflow/python/eager:run_eager_op_as_function_xla_test_cpu PASSED in 7.1s //tensorflow/python/eager:small_constants_optimizer_test_cpu PASSED in 156.0s //tensorflow/python/eager:tensor_test_cpu PASSED in 11.7s //tensorflow/python/eager:wrap_function_device_test_cpu PASSED in 8.7s //tensorflow/python/eager:wrap_function_test PASSED in 10.3s //tensorflow/python/eager/benchmarks:kpi_benchmark_test_cpu PASSED in 33.5s //tensorflow/python/eager/memory_tests:remote_memory_test_cpu PASSED in 8.3s //tensorflow/python/eager/polymorphic_function:argument_naming_test_cpu PASSED in 9.2s //tensorflow/python/eager/polymorphic_function:atomic_function_test_cpu PASSED in 22.1s //tensorflow/python/eager/polymorphic_function:collection_test_cpu PASSED in 10.3s //tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu PASSED in 9.9s //tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu_mlir_bridge_test PASSED in 19.5s //tensorflow/python/eager/polymorphic_function:function_spec_test PASSED in 7.7s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_jit_test_cpu PASSED in 41.6s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_jit_test_cpu_mlir_bridge_test PASSED in 40.0s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_test_cpu PASSED in 8.0s //tensorflow/python/eager/polymorphic_function:quarantine_test PASSED in 33.1s //tensorflow/python/feature_column:sequence_feature_column_integration_test PASSED in 8.5s //tensorflow/python/feature_column:serialization_test PASSED in 11.6s //tensorflow/python/framework:auto_control_deps_test PASSED in 41.6s //tensorflow/python/framework:c_api_util_test PASSED in 22.3s //tensorflow/python/framework:common_shapes_test PASSED in 10.3s //tensorflow/python/framework:composite_tensor_test PASSED in 11.5s //tensorflow/python/framework:config_test_2gpu PASSED in 12.8s //tensorflow/python/framework:config_test_cpu PASSED in 12.5s //tensorflow/python/framework:constant_op_test PASSED in 18.4s //tensorflow/python/framework:device_spec_test PASSED in 8.2s //tensorflow/python/framework:device_test PASSED in 7.2s //tensorflow/python/framework:dtypes_test PASSED in 15.0s //tensorflow/python/framework:error_interpolation_test PASSED in 9.7s //tensorflow/python/framework:errors_test PASSED in 8.0s //tensorflow/python/framework:extension_type_field_test PASSED in 8.0s //tensorflow/python/framework:extension_type_test PASSED in 15.6s //tensorflow/python/framework:file_system_test PASSED in 8.4s //tensorflow/python/framework:function_def_to_graph_test PASSED in 9.5s //tensorflow/python/framework:graph_building_benchmark_cpu PASSED in 8.0s //tensorflow/python/framework:graph_util_test PASSED in 25.9s //tensorflow/python/framework:immutable_dict_test PASSED in 8.6s //tensorflow/python/framework:importer_test PASSED in 8.6s //tensorflow/python/framework:indexed_slices_test PASSED in 8.9s //tensorflow/python/framework:kernels_test PASSED in 7.2s //tensorflow/python/framework:meta_graph_test PASSED in 8.9s //tensorflow/python/framework:node_file_writer_test_cpu PASSED in 8.2s //tensorflow/python/framework:offset_counter_helper_test PASSED in 0.2s //tensorflow/python/framework:op_allowlist_namespace_test PASSED in 4.7s //tensorflow/python/framework:op_callbacks_test_cpu PASSED in 11.4s //tensorflow/python/framework:op_def_library_test PASSED in 22.1s //tensorflow/python/framework:op_def_util_test PASSED in 7.4s //tensorflow/python/framework:ops_enable_eager_test PASSED in 2.4s //tensorflow/python/framework:ops_test PASSED in 23.2s //tensorflow/python/framework:proto_test PASSED in 27.1s //tensorflow/python/framework:py_context_manager_test PASSED in 17.4s //tensorflow/python/framework:python_api_dispatcher_test PASSED in 9.9s //tensorflow/python/framework:python_api_info_test PASSED in 8.7s //tensorflow/python/framework:python_api_parameter_converter_test PASSED in 7.8s //tensorflow/python/framework:python_op_gen_annotation_test PASSED in 3.4s //tensorflow/python/framework:python_op_gen_annotator_test PASSED in 0.1s //tensorflow/python/framework:python_tensor_converter_test PASSED in 29.1s //tensorflow/python/framework:random_seed_test PASSED in 16.6s //tensorflow/python/framework:registry_test PASSED in 6.7s //tensorflow/python/framework:smart_cond_test PASSED in 11.2s //tensorflow/python/framework:sparse_tensor_test PASSED in 12.0s //tensorflow/python/framework:subscribe_test PASSED in 7.2s //tensorflow/python/framework:tensor_shape_test PASSED in 7.8s //tensorflow/python/framework:tensor_test PASSED in 7.5s //tensorflow/python/framework:tensor_util_test PASSED in 19.2s //tensorflow/python/framework:test_combinations_test PASSED in 6.0s //tensorflow/python/framework:test_util_test_cpu PASSED in 20.0s //tensorflow/python/framework:tf2_test PASSED in 6.8s //tensorflow/python/framework:traceable_stack_test PASSED in 11.5s //tensorflow/python/framework:type_spec_test PASSED in 7.8s //tensorflow/python/framework:versions_test PASSED in 8.1s //tensorflow/python/framework/experimental:graph_building_test_cpu PASSED in 6.9s //tensorflow/python/framework/experimental:unified_api_test_cpu PASSED in 13.1s //tensorflow/python/grappler:arithmetic_optimizer_test_cpu PASSED in 8.5s //tensorflow/python/grappler:auto_mixed_precision_test_cpu PASSED in 16.8s //tensorflow/python/grappler:constant_folding_test_cpu PASSED in 9.5s //tensorflow/python/grappler:cost_analyzer_test PASSED in 14.3s //tensorflow/python/grappler:datasets_test PASSED in 9.2s //tensorflow/python/grappler:item_test PASSED in 7.9s //tensorflow/python/grappler:memory_optimizer_test PASSED in 23.1s //tensorflow/python/grappler:model_analyzer_test PASSED in 6.6s //tensorflow/python/grappler:remapper_test_cpu PASSED in 8.0s //tensorflow/python/grappler:tf_optimizer_test PASSED in 10.4s //tensorflow/python/kernel_tests:benchmark_test_cpu PASSED in 10.4s //tensorflow/python/kernel_tests:check_ops_test_cpu PASSED in 16.2s //tensorflow/python/kernel_tests:collective_ops_multi_worker_test PASSED in 35.3s //tensorflow/python/kernel_tests:composite_tensor_ops_test PASSED in 9.1s //tensorflow/python/kernel_tests:critical_section_test_cpu PASSED in 20.9s //tensorflow/python/kernel_tests:garbage_collection_test PASSED in 8.2s //tensorflow/python/kernel_tests:gradient_correctness_test_cpu PASSED in 13.0s //tensorflow/python/kernel_tests:histogram_ops_test_cpu PASSED in 7.2s //tensorflow/python/kernel_tests:logging_ops_test_cpu PASSED in 9.4s //tensorflow/python/kernel_tests:numerics_test_cpu PASSED in 7.6s //tensorflow/python/kernel_tests:template_test PASSED in 9.1s //tensorflow/python/kernel_tests:trace_op_test_cpu PASSED in 6.9s //tensorflow/python/kernel_tests/array_ops:batch_gather_op_test_cpu PASSED in 8.4s //tensorflow/python/kernel_tests/array_ops:batch_scatter_ops_test PASSED in 9.2s //tensorflow/python/kernel_tests/array_ops:batchtospace_op_test_cpu PASSED in 11.3s //tensorflow/python/kernel_tests/array_ops:bcast_ops_test PASSED in 8.3s //tensorflow/python/kernel_tests/array_ops:bitcast_op_test_cpu PASSED in 7.0s //tensorflow/python/kernel_tests/array_ops:broadcast_to_ops_test_cpu PASSED in 34.9s //tensorflow/python/kernel_tests/array_ops:cast_op_test_cpu PASSED in 8.5s //tensorflow/python/kernel_tests/array_ops:constant_op_eager_test_cpu PASSED in 8.5s //tensorflow/python/kernel_tests/array_ops:constant_op_test_cpu PASSED in 12.2s //tensorflow/python/kernel_tests/array_ops:denormal_test_cpu PASSED in 7.9s //tensorflow/python/kernel_tests/array_ops:depthtospace_op_test_cpu PASSED in 13.0s //tensorflow/python/kernel_tests/array_ops:edit_distance_op_test PASSED in 8.7s //tensorflow/python/kernel_tests/array_ops:fingerprint_op_test PASSED in 7.7s //tensorflow/python/kernel_tests/array_ops:gather_nd_op_test_cpu PASSED in 8.8s //tensorflow/python/kernel_tests/array_ops:identity_n_op_py_test PASSED in 10.2s //tensorflow/python/kernel_tests/array_ops:identity_op_py_test PASSED in 7.3s //tensorflow/python/kernel_tests/array_ops:large_concat_op_test_cpu PASSED in 7.7s //tensorflow/python/kernel_tests/array_ops:manip_ops_test_cpu PASSED in 8.0s //tensorflow/python/kernel_tests/array_ops:one_hot_op_test_cpu PASSED in 7.8s //tensorflow/python/kernel_tests/array_ops:pad_op_test_cpu PASSED in 16.5s //tensorflow/python/kernel_tests/array_ops:reshape_op_test_cpu PASSED in 9.9s //tensorflow/python/kernel_tests/array_ops:reverse_sequence_op_test_cpu PASSED in 7.3s //tensorflow/python/kernel_tests/array_ops:scalar_test_cpu PASSED in 9.0s //tensorflow/python/kernel_tests/array_ops:shape_ops_test_cpu PASSED in 13.7s //tensorflow/python/kernel_tests/array_ops:slice_op_test_cpu PASSED in 9.0s //tensorflow/python/kernel_tests/array_ops:spacetobatch_op_test_cpu PASSED in 13.9s //tensorflow/python/kernel_tests/array_ops:spacetodepth_op_test_cpu PASSED in 7.9s //tensorflow/python/kernel_tests/array_ops:stack_op_test_cpu PASSED in 15.4s //tensorflow/python/kernel_tests/array_ops:unique_op_test_cpu PASSED in 8.3s //tensorflow/python/kernel_tests/array_ops:unstack_op_test_cpu PASSED in 12.3s //tensorflow/python/kernel_tests/array_ops:where_op_test_cpu PASSED in 15.5s //tensorflow/python/kernel_tests/control_flow:cond_v2_test_cpu PASSED in 50.3s //tensorflow/python/kernel_tests/control_flow:control_flow_util_test PASSED in 7.4s //tensorflow/python/kernel_tests/control_flow:control_flow_util_v2_test PASSED in 20.3s //tensorflow/python/kernel_tests/control_flow:py_func_test_cpu PASSED in 17.3s //tensorflow/python/kernel_tests/control_flow:scan_ops_test_cpu PASSED in 67.9s //tensorflow/python/kernel_tests/control_flow:while_v2_test_cpu PASSED in 60.0s //tensorflow/python/kernel_tests/custom_ops:ackermann_test PASSED in 7.6s //tensorflow/python/kernel_tests/custom_ops:duplicate_op_test PASSED in 7.4s //tensorflow/python/kernel_tests/custom_ops:invalid_op_test PASSED in 22.1s //tensorflow/python/kernel_tests/data_structures:conditional_accumulator_test PASSED in 14.1s //tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_2gpu PASSED in 12.3s //tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_cpu PASSED in 14.5s //tensorflow/python/kernel_tests/data_structures:dynamic_stitch_op_test_cpu PASSED in 7.4s //tensorflow/python/kernel_tests/data_structures:fifo_queue_test PASSED in 9.8s //tensorflow/python/kernel_tests/data_structures:list_ops_test_cpu PASSED in 22.8s //tensorflow/python/kernel_tests/data_structures:listdiff_op_test PASSED in 9.8s //tensorflow/python/kernel_tests/data_structures:lookup_ops_test PASSED in 20.1s //tensorflow/python/kernel_tests/data_structures:map_ops_test PASSED in 27.7s //tensorflow/python/kernel_tests/data_structures:padding_fifo_queue_test_cpu PASSED in 7.6s //tensorflow/python/kernel_tests/data_structures:priority_queue_test PASSED in 7.3s //tensorflow/python/kernel_tests/data_structures:stack_ops_test_cpu PASSED in 7.5s //tensorflow/python/kernel_tests/data_structures:stage_op_test_cpu PASSED in 8.7s //tensorflow/python/kernel_tests/distributions:bernoulli_test_cpu PASSED in 16.5s //tensorflow/python/kernel_tests/distributions:bijector_test_cpu PASSED in 8.1s //tensorflow/python/kernel_tests/distributions:categorical_test_cpu PASSED in 9.1s //tensorflow/python/kernel_tests/distributions:dirichlet_multinomial_test_cpu PASSED in 13.1s //tensorflow/python/kernel_tests/distributions:dirichlet_test_cpu PASSED in 15.2s //tensorflow/python/kernel_tests/distributions:exponential_test_cpu PASSED in 10.3s //tensorflow/python/kernel_tests/distributions:gamma_test_cpu PASSED in 52.0s //tensorflow/python/kernel_tests/distributions:identity_bijector_test_cpu PASSED in 8.8s //tensorflow/python/kernel_tests/distributions:kullback_leibler_test_cpu PASSED in 8.1s //tensorflow/python/kernel_tests/distributions:laplace_test_cpu PASSED in 35.1s //tensorflow/python/kernel_tests/distributions:multinomial_test_cpu PASSED in 8.4s //tensorflow/python/kernel_tests/distributions:normal_test_cpu PASSED in 21.4s //tensorflow/python/kernel_tests/distributions:special_math_test_cpu PASSED in 17.9s //tensorflow/python/kernel_tests/distributions:uniform_test_cpu PASSED in 11.1s //tensorflow/python/kernel_tests/image_ops:attention_ops_test PASSED in 7.9s //tensorflow/python/kernel_tests/image_ops:decode_bmp_op_test PASSED in 7.5s //tensorflow/python/kernel_tests/image_ops:decode_compressed_op_test PASSED in 10.4s //tensorflow/python/kernel_tests/image_ops:decode_image_op_test PASSED in 7.8s //tensorflow/python/kernel_tests/image_ops:decode_jpeg_op_test PASSED in 8.0s //tensorflow/python/kernel_tests/image_ops:decode_png_op_test PASSED in 7.8s //tensorflow/python/kernel_tests/image_ops:decode_raw_op_test PASSED in 7.9s //tensorflow/python/kernel_tests/image_ops:draw_bounding_box_op_test_cpu PASSED in 7.8s //tensorflow/python/kernel_tests/image_ops:extract_image_patches_op_test_cpu PASSED in 6.5s //tensorflow/python/kernel_tests/image_ops:extract_volume_patches_op_test_cpu PASSED in 8.2s //tensorflow/python/kernel_tests/io_ops:checkpoint_ops_test PASSED in 10.0s //tensorflow/python/kernel_tests/io_ops:decode_csv_op_test PASSED in 7.4s //tensorflow/python/kernel_tests/io_ops:io_ops_test PASSED in 8.8s //tensorflow/python/kernel_tests/io_ops:parse_single_example_op_test PASSED in 11.1s //tensorflow/python/kernel_tests/io_ops:parsing_ops_test PASSED in 30.0s //tensorflow/python/kernel_tests/io_ops:reader_ops_test PASSED in 8.7s //tensorflow/python/kernel_tests/io_ops:record_input_test PASSED in 28.1s //tensorflow/python/kernel_tests/io_ops:save_restore_ops_test PASSED in 7.9s //tensorflow/python/kernel_tests/linalg:determinant_op_test_cpu PASSED in 7.6s //tensorflow/python/kernel_tests/linalg:linear_operator_addition_test_cpu PASSED in 9.8s //tensorflow/python/kernel_tests/linalg:linear_operator_algebra_test_cpu PASSED in 7.1s //tensorflow/python/kernel_tests/linalg:linear_operator_test_cpu PASSED in 22.0s //tensorflow/python/kernel_tests/linalg:lu_op_test_cpu PASSED in 9.8s //tensorflow/python/kernel_tests/linalg:matrix_inverse_op_test_cpu PASSED in 8.4s //tensorflow/python/kernel_tests/linalg:matrix_logarithm_op_test PASSED in 61.9s //tensorflow/python/kernel_tests/linalg:matrix_solve_ls_op_test_cpu PASSED in 19.9s //tensorflow/python/kernel_tests/linalg:matrix_solve_op_test_cpu PASSED in 17.4s //tensorflow/python/kernel_tests/linalg:matrix_square_root_op_test_cpu PASSED in 6.6s //tensorflow/python/kernel_tests/linalg:slicing_test_cpu PASSED in 10.2s //tensorflow/python/kernel_tests/linalg/sparse:conjugate_gradient_test_cpu PASSED in 19.9s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_test_cpu PASSED in 7.0s //tensorflow/python/kernel_tests/math_ops:aggregate_ops_test_cpu PASSED in 9.0s //tensorflow/python/kernel_tests/math_ops:argmax_op_test_cpu PASSED in 17.3s //tensorflow/python/kernel_tests/math_ops:banded_triangular_solve_op_test_cpu PASSED in 9.9s //tensorflow/python/kernel_tests/math_ops:basic_gpu_test_cpu PASSED in 7.8s //tensorflow/python/kernel_tests/math_ops:bincount_op_test_cpu PASSED in 18.1s //tensorflow/python/kernel_tests/math_ops:bucketize_op_test_cpu PASSED in 7.6s //tensorflow/python/kernel_tests/math_ops:clip_ops_test PASSED in 10.5s //tensorflow/python/kernel_tests/math_ops:confusion_matrix_test PASSED in 23.7s //tensorflow/python/kernel_tests/math_ops:cross_grad_test_cpu PASSED in 6.8s //tensorflow/python/kernel_tests/math_ops:cumulative_logsumexp_test_cpu PASSED in 8.7s //tensorflow/python/kernel_tests/math_ops:in_topk_op_test_cpu PASSED in 7.0s //tensorflow/python/kernel_tests/math_ops:reduce_benchmark_test_cpu PASSED in 19.1s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_d9m_test_cpu PASSED in 7.8s //tensorflow/python/kernel_tests/math_ops:sets_test PASSED in 31.4s //tensorflow/python/kernel_tests/math_ops:topk_op_test_cpu PASSED in 21.7s //tensorflow/python/kernel_tests/math_ops:zero_division_test_cpu PASSED in 6.9s //tensorflow/python/kernel_tests/nn_ops:betainc_op_test_cpu PASSED in 15.4s //tensorflow/python/kernel_tests/nn_ops:bias_op_test_cpu PASSED in 138.0s //tensorflow/python/kernel_tests/nn_ops:conv1d_test_cpu PASSED in 8.1s //tensorflow/python/kernel_tests/nn_ops:conv1d_transpose_test_cpu PASSED in 7.3s //tensorflow/python/kernel_tests/nn_ops:conv2d_transpose_test_cpu PASSED in 11.1s //tensorflow/python/kernel_tests/nn_ops:conv3d_backprop_filter_v2_grad_test_cpu PASSED in 11.4s //tensorflow/python/kernel_tests/nn_ops:conv3d_transpose_test_cpu PASSED in 8.1s //tensorflow/python/kernel_tests/nn_ops:ctc_decoder_ops_test PASSED in 7.9s //tensorflow/python/kernel_tests/nn_ops:ctc_loss_op_test_cpu PASSED in 61.9s //tensorflow/python/kernel_tests/nn_ops:cudnn_d9m_test_cpu PASSED in 7.7s //tensorflow/python/kernel_tests/nn_ops:cudnn_deterministic_ops_test_cpu PASSED in 7.5s //tensorflow/python/kernel_tests/nn_ops:losses_test PASSED in 31.4s //tensorflow/python/kernel_tests/nn_ops:lrn_op_test_cpu PASSED in 15.4s //tensorflow/python/kernel_tests/nn_ops:morphological_ops_test_cpu PASSED in 11.9s //tensorflow/python/kernel_tests/nn_ops:nth_element_op_test_cpu PASSED in 8.2s //tensorflow/python/kernel_tests/nn_ops:pool_test_cpu PASSED in 28.7s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_3d_test_cpu PASSED in 17.9s //tensorflow/python/kernel_tests/nn_ops:relu_op_test_cpu PASSED in 16.9s //tensorflow/python/kernel_tests/nn_ops:softmax_op_test_cpu PASSED in 7.5s //tensorflow/python/kernel_tests/nn_ops:softplus_op_test_cpu PASSED in 8.0s //tensorflow/python/kernel_tests/nn_ops:softsign_op_test_cpu PASSED in 6.9s //tensorflow/python/kernel_tests/nn_ops:xent_op_d9m_test_cpu PASSED in 153.8s //tensorflow/python/kernel_tests/nn_ops:xent_op_test_cpu PASSED in 10.5s //tensorflow/python/kernel_tests/proto:descriptor_source_test PASSED in 7.3s //tensorflow/python/kernel_tests/proto:encode_proto_op_test PASSED in 8.5s //tensorflow/python/kernel_tests/quantization_ops:quantization_ops_test PASSED in 8.1s //tensorflow/python/kernel_tests/random:candidate_sampler_ops_test PASSED in 14.3s //tensorflow/python/kernel_tests/random:multinomial_op_test_cpu PASSED in 11.2s //tensorflow/python/kernel_tests/random:parameterized_truncated_normal_op_test_cpu PASSED in 15.2s //tensorflow/python/kernel_tests/random:random_crop_test_cpu PASSED in 16.3s //tensorflow/python/kernel_tests/random:random_grad_test_cpu PASSED in 10.3s //tensorflow/python/kernel_tests/random:random_ops_test_cpu PASSED in 13.8s //tensorflow/python/kernel_tests/random:random_poisson_test_cpu PASSED in 9.1s //tensorflow/python/kernel_tests/random:random_shuffle_queue_test PASSED in 16.4s //tensorflow/python/kernel_tests/random:stateful_random_ops_test_cpu PASSED in 40.2s //tensorflow/python/kernel_tests/signal:mel_ops_test_cpu PASSED in 13.1s //tensorflow/python/kernel_tests/signal:mfcc_ops_test_cpu PASSED in 20.1s //tensorflow/python/kernel_tests/signal:reconstruction_ops_test_cpu PASSED in 10.4s //tensorflow/python/kernel_tests/signal:shape_ops_test_cpu PASSED in 30.2s //tensorflow/python/kernel_tests/sparse_ops:sparse_add_op_test PASSED in 10.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_concat_op_test PASSED in 18.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_conditional_accumulator_test PASSED in 18.1s //tensorflow/python/kernel_tests/sparse_ops:sparse_cross_op_test PASSED in 24.2s //tensorflow/python/kernel_tests/sparse_ops:sparse_matmul_op_test_cpu PASSED in 47.2s //tensorflow/python/kernel_tests/sparse_ops:sparse_reorder_op_test PASSED in 22.2s //tensorflow/python/kernel_tests/sparse_ops:sparse_reshape_op_test PASSED in 9.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_serialization_ops_test PASSED in 9.1s //tensorflow/python/kernel_tests/sparse_ops:sparse_slice_op_test PASSED in 9.2s //tensorflow/python/kernel_tests/sparse_ops:sparse_split_op_test_cpu PASSED in 8.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_grad_test_cpu PASSED in 15.4s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_d9m_test_cpu PASSED in 43.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_test_cpu PASSED in 26.4s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensors_map_ops_test PASSED in 16.2s //tensorflow/python/kernel_tests/sparse_ops:sparse_to_dense_op_py_test_cpu PASSED in 7.8s //tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_d9m_test_cpu PASSED in 76.9s //tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_test_cpu PASSED in 8.5s //tensorflow/python/kernel_tests/sparse_ops:sparsemask_op_test PASSED in 8.0s //tensorflow/python/kernel_tests/strings_ops:as_string_op_test PASSED in 7.5s //tensorflow/python/kernel_tests/strings_ops:base64_ops_test PASSED in 13.4s //tensorflow/python/kernel_tests/strings_ops:reduce_join_op_test_cpu PASSED in 8.5s //tensorflow/python/kernel_tests/strings_ops:regex_full_match_op_test PASSED in 9.4s //tensorflow/python/kernel_tests/strings_ops:regex_replace_op_test PASSED in 8.6s //tensorflow/python/kernel_tests/strings_ops:string_bytes_split_op_test PASSED in 8.2s //tensorflow/python/kernel_tests/strings_ops:string_format_op_test PASSED in 8.6s //tensorflow/python/kernel_tests/strings_ops:string_join_op_test PASSED in 7.9s //tensorflow/python/kernel_tests/strings_ops:string_length_op_test PASSED in 8.6s //tensorflow/python/kernel_tests/strings_ops:string_lower_op_test PASSED in 7.6s //tensorflow/python/kernel_tests/strings_ops:string_split_op_test PASSED in 9.9s //tensorflow/python/kernel_tests/strings_ops:string_strip_op_test PASSED in 7.8s //tensorflow/python/kernel_tests/strings_ops:string_to_hash_bucket_op_test_cpu PASSED in 7.5s //tensorflow/python/kernel_tests/strings_ops:string_to_number_op_test_cpu PASSED in 7.7s //tensorflow/python/kernel_tests/strings_ops:string_upper_op_test PASSED in 11.1s //tensorflow/python/kernel_tests/strings_ops:substr_op_test PASSED in 9.5s //tensorflow/python/kernel_tests/strings_ops:unicode_decode_op_test PASSED in 17.8s //tensorflow/python/kernel_tests/strings_ops:unicode_encode_op_test PASSED in 7.6s //tensorflow/python/kernel_tests/strings_ops:unicode_script_op_test PASSED in 7.2s //tensorflow/python/kernel_tests/strings_ops:unicode_transcode_op_test PASSED in 7.8s //tensorflow/python/kernel_tests/strings_ops:unsorted_segment_join_op_test_cpu PASSED in 8.1s //tensorflow/python/kernel_tests/summary_ops:summary_ops_test_cpu PASSED in 15.2s //tensorflow/python/kernel_tests/summary_ops:summary_v1_audio_op_test_cpu PASSED in 7.5s //tensorflow/python/kernel_tests/summary_ops:summary_v1_image_op_test_cpu PASSED in 8.4s //tensorflow/python/kernel_tests/summary_ops:summary_v1_ops_test PASSED in 7.5s //tensorflow/python/kernel_tests/summary_ops:summary_v1_tensor_op_test PASSED in 7.9s //tensorflow/python/kernel_tests/v1_compat_tests:array_ops_test_cpu PASSED in 8.8s //tensorflow/python/kernel_tests/v1_compat_tests:dense_update_ops_test_cpu PASSED in 7.8s //tensorflow/python/kernel_tests/v1_compat_tests:identity_op_py_test PASSED in 16.3s //tensorflow/python/kernel_tests/v1_compat_tests:scatter_nd_ops_test_cpu PASSED in 9.0s //tensorflow/python/kernel_tests/v1_compat_tests:session_ops_test_cpu PASSED in 8.1s //tensorflow/python/kernel_tests/v1_compat_tests:stack_op_test_cpu PASSED in 12.1s //tensorflow/python/kernel_tests/variables:dense_update_ops_no_tsan_test_cpu PASSED in 7.5s //tensorflow/python/kernel_tests/variables:dense_update_ops_test_cpu PASSED in 7.4s //tensorflow/python/kernel_tests/variables:partitioned_variables_test PASSED in 11.2s //tensorflow/python/kernel_tests/variables:resource_variable_ops_test_cpu PASSED in 53.4s //tensorflow/python/kernel_tests/variables:variable_ops_test_cpu PASSED in 8.8s //tensorflow/python/kernel_tests/variables:variable_scope_test PASSED in 50.7s //tensorflow/python/kernel_tests/variables:variables_test PASSED in 9.1s //tensorflow/python/lib/core:custom_float_test PASSED in 7.5s //tensorflow/python/lib/io:file_io_test PASSED in 24.9s //tensorflow/python/lib/io:tf_record_test PASSED in 9.7s //tensorflow/python/module:module_test PASSED in 8.3s //tensorflow/python/ops/losses:util_test PASSED in 6.7s //tensorflow/python/ops/memory_tests:custom_gradient_memory_test_cpu PASSED in 13.0s //tensorflow/python/ops/numpy_ops:np_array_ops_test_cpu PASSED in 69.6s //tensorflow/python/ops/numpy_ops:np_arrays_test_cpu PASSED in 8.4s //tensorflow/python/ops/numpy_ops:np_dtypes_test_cpu PASSED in 7.3s //tensorflow/python/ops/numpy_ops:np_interop_test_cpu PASSED in 39.9s //tensorflow/python/ops/numpy_ops:np_logic_test_cpu PASSED in 10.4s //tensorflow/python/ops/numpy_ops:np_math_ops_test_cpu PASSED in 22.5s //tensorflow/python/ops/numpy_ops:np_random_test_cpu PASSED in 66.2s //tensorflow/python/ops/numpy_ops:np_utils_test_cpu PASSED in 7.3s //tensorflow/python/ops/numpy_ops/integration_test:np_config_test_cpu PASSED in 20.2s //tensorflow/python/ops/numpy_ops/integration_test:public_symbol_test PASSED in 18.9s //tensorflow/python/ops/parallel_for:array_test_cpu PASSED in 36.0s //tensorflow/python/ops/parallel_for:gradients_test_cpu PASSED in 11.6s //tensorflow/python/ops/parallel_for:xla_control_flow_ops_test_cpu PASSED in 54.0s //tensorflow/python/ops/ragged:convert_to_tensor_or_ragged_tensor_op_test PASSED in 8.0s //tensorflow/python/ops/ragged:ragged_batch_gather_op_test PASSED in 44.3s //tensorflow/python/ops/ragged:ragged_bitcast_op_test PASSED in 7.6s //tensorflow/python/ops/ragged:ragged_boolean_mask_op_test PASSED in 14.9s //tensorflow/python/ops/ragged:ragged_concat_op_test PASSED in 11.5s //tensorflow/python/ops/ragged:ragged_const_op_test PASSED in 8.4s //tensorflow/python/ops/ragged:ragged_constant_value_op_test PASSED in 7.2s //tensorflow/python/ops/ragged:ragged_cross_op_test PASSED in 20.7s //tensorflow/python/ops/ragged:ragged_dispatch_test PASSED in 122.2s //tensorflow/python/ops/ragged:ragged_dynamic_partition_op_test_cpu PASSED in 17.1s //tensorflow/python/ops/ragged:ragged_eager_test PASSED in 9.0s //tensorflow/python/ops/ragged:ragged_expand_dims_op_test PASSED in 8.4s //tensorflow/python/ops/ragged:ragged_factory_ops_test_cpu PASSED in 13.1s //tensorflow/python/ops/ragged:ragged_fill_empty_rows_op_test PASSED in 8.7s //tensorflow/python/ops/ragged:ragged_from_sparse_op_test PASSED in 8.6s //tensorflow/python/ops/ragged:ragged_from_tensor_op_test PASSED in 21.2s //tensorflow/python/ops/ragged:ragged_gather_nd_op_test PASSED in 10.0s //tensorflow/python/ops/ragged:ragged_map_flat_values_op_test PASSED in 8.4s //tensorflow/python/ops/ragged:ragged_map_fn_op_test PASSED in 18.3s //tensorflow/python/ops/ragged:ragged_math_ops_test PASSED in 12.9s //tensorflow/python/ops/ragged:ragged_matmul_op_test PASSED in 34.1s //tensorflow/python/ops/ragged:ragged_merge_dims_op_test PASSED in 25.9s //tensorflow/python/ops/ragged:ragged_one_hot_op_test PASSED in 9.0s //tensorflow/python/ops/ragged:ragged_operators_test PASSED in 20.3s //tensorflow/python/ops/ragged:ragged_placeholder_op_test PASSED in 6.8s //tensorflow/python/ops/ragged:ragged_print_op_test PASSED in 13.9s //tensorflow/python/ops/ragged:ragged_range_op_test PASSED in 7.4s //tensorflow/python/ops/ragged:ragged_rank_op_test PASSED in 8.0s //tensorflow/python/ops/ragged:ragged_reduce_op_test PASSED in 30.0s //tensorflow/python/ops/ragged:ragged_resize_image_op_test PASSED in 17.4s //tensorflow/python/ops/ragged:ragged_reverse_op_test PASSED in 7.3s //tensorflow/python/ops/ragged:ragged_row_lengths_op_test PASSED in 11.2s //tensorflow/python/ops/ragged:ragged_row_splits_to_segment_ids_op_test PASSED in 8.4s //tensorflow/python/ops/ragged:ragged_segment_ids_to_row_splits_op_test PASSED in 9.3s //tensorflow/python/ops/ragged:ragged_segment_op_test PASSED in 17.3s //tensorflow/python/ops/ragged:ragged_size_op_test PASSED in 7.6s //tensorflow/python/ops/ragged:ragged_split_op_test PASSED in 41.5s //tensorflow/python/ops/ragged:ragged_squeeze_op_test PASSED in 17.8s //tensorflow/python/ops/ragged:ragged_stack_op_test PASSED in 12.9s //tensorflow/python/ops/ragged:ragged_tensor_bounding_shape_op_test PASSED in 8.0s //tensorflow/python/ops/ragged:ragged_tensor_shape_test PASSED in 60.1s //tensorflow/python/ops/ragged:ragged_tile_op_test PASSED in 36.1s //tensorflow/python/ops/ragged:ragged_to_sparse_op_test PASSED in 7.2s //tensorflow/python/ops/ragged:ragged_to_tensor_op_test PASSED in 67.9s //tensorflow/python/ops/ragged:ragged_util_test PASSED in 21.5s //tensorflow/python/ops/ragged:ragged_where_op_test PASSED in 28.2s //tensorflow/python/ops/ragged:row_partition_test PASSED in 39.9s //tensorflow/python/ops/ragged:string_ngrams_op_test PASSED in 7.4s //tensorflow/python/ops/ragged:strings_reduce_join_op_test PASSED in 8.2s //tensorflow/python/ops/structured:structured_array_ops_test PASSED in 41.6s //tensorflow/python/ops/structured:structured_tensor_slice_test PASSED in 63.2s //tensorflow/python/ops/structured:structured_tensor_spec_test PASSED in 11.5s //tensorflow/python/ops/structured:structured_tensor_test PASSED in 40.5s //tensorflow/python/ops/v1_compat_tests:gradient_checker_test_cpu PASSED in 8.7s //tensorflow/python/platform:benchmark_test PASSED in 7.2s //tensorflow/python/platform:build_info_test PASSED in 7.6s //tensorflow/python/platform:resource_loader_test PASSED in 1.9s //tensorflow/python/profiler:pprof_profiler_test PASSED in 6.6s //tensorflow/python/profiler:profile_context_test_cpu PASSED in 26.7s //tensorflow/python/profiler:profiler_client_test_cpu PASSED in 7.8s //tensorflow/python/profiler:profiler_test_cpu PASSED in 18.8s //tensorflow/python/profiler:profiler_v2_test_cpu PASSED in 9.6s //tensorflow/python/profiler:profiler_wrapper_test PASSED in 6.8s //tensorflow/python/profiler:tfprof_logger_test PASSED in 7.1s //tensorflow/python/profiler/integration_test:profiler_api_test_cpu PASSED in 29.3s //tensorflow/python/profiler/internal:flops_registry_test PASSED in 6.8s //tensorflow/python/profiler/internal:print_model_analysis_test PASSED in 6.4s //tensorflow/python/profiler/internal:run_metadata_test_cpu PASSED in 12.0s //tensorflow/python/saved_model:fingerprinting_test PASSED in 9.2s //tensorflow/python/saved_model:keras_injection_test PASSED in 14.4s //tensorflow/python/saved_model:load_v1_in_v2_test PASSED in 14.1s //tensorflow/python/saved_model:loader_test PASSED in 10.7s //tensorflow/python/saved_model:method_name_updater_test PASSED in 6.6s //tensorflow/python/saved_model:metrics_test PASSED in 8.7s //tensorflow/python/saved_model:nested_structure_coder_test PASSED in 8.7s //tensorflow/python/saved_model:pywrap_saved_model_fingerprinting_test PASSED in 8.0s //tensorflow/python/saved_model:pywrap_saved_model_metrics_test PASSED in 6.8s //tensorflow/python/saved_model:revived_types_test PASSED in 7.0s //tensorflow/python/saved_model:save_context_test PASSED in 6.8s //tensorflow/python/saved_model:save_test PASSED in 33.7s //tensorflow/python/saved_model:saved_model_test PASSED in 18.8s //tensorflow/python/saved_model:signature_def_utils_test PASSED in 8.0s //tensorflow/python/saved_model:simple_save_test PASSED in 7.9s //tensorflow/python/saved_model:tracing_utils_test PASSED in 8.4s //tensorflow/python/saved_model:utils_test PASSED in 8.2s //tensorflow/python/saved_model/model_utils:export_output_test PASSED in 18.2s //tensorflow/python/saved_model/model_utils:export_test PASSED in 14.7s //tensorflow/python/saved_model/model_utils:mode_keys_test PASSED in 6.7s //tensorflow/python/saved_model/registration:registration_saving_test PASSED in 14.2s //tensorflow/python/saved_model/registration:registration_test PASSED in 7.5s //tensorflow/python/saved_model/registration:tf_registration_test PASSED in 13.8s //tensorflow/python/summary:plugin_asset_test PASSED in 8.0s //tensorflow/python/summary:summary_iterator_test PASSED in 7.6s //tensorflow/python/summary:summary_test PASSED in 8.3s //tensorflow/python/summary:summary_v2_test PASSED in 7.1s //tensorflow/python/summary/writer:writer_test PASSED in 19.3s //tensorflow/python/tools:aot_compiled_test PASSED in 21.7s //tensorflow/python/tools:freeze_graph_test PASSED in 21.7s //tensorflow/python/tools:optimize_for_inference_test PASSED in 8.1s //tensorflow/python/tools:print_selective_registration_header_test PASSED in 45.6s //tensorflow/python/tools:saved_model_cli_test PASSED in 32.0s //tensorflow/python/tools:saved_model_utils_test PASSED in 7.8s //tensorflow/python/tools:strip_unused_test PASSED in 7.4s //tensorflow/python/tools/api/generator:create_python_api_test PASSED in 37.2s //tensorflow/python/tools/api/generator:output_init_files_test PASSED in 14.3s //tensorflow/python/tools/api/generator:tensorflow_doc_srcs_test PASSED in 11.6s //tensorflow/python/tpu:bfloat16_test PASSED in 7.5s //tensorflow/python/tpu:feature_column_test PASSED in 25.2s //tensorflow/python/tpu:topology_test PASSED in 6.6s //tensorflow/python/tpu:tpu_embedding_for_serving_test PASSED in 23.5s //tensorflow/python/tpu:tpu_embedding_v2_utils_test PASSED in 9.8s //tensorflow/python/tpu:tpu_infeed_test PASSED in 31.2s //tensorflow/python/tpu:tpu_sharding_test PASSED in 7.3s //tensorflow/python/tpu:tpu_test_wrapper_test PASSED in 7.3s //tensorflow/python/tpu/client:client_py_test PASSED in 10.1s //tensorflow/python/trackable:autotrackable_test PASSED in 22.4s //tensorflow/python/trackable:base_delegate_test PASSED in 8.8s //tensorflow/python/trackable:base_test PASSED in 8.0s //tensorflow/python/trackable:data_structures_test PASSED in 12.7s //tensorflow/python/trackable:python_state_test PASSED in 8.2s //tensorflow/python/trackable:resource_test PASSED in 7.7s //tensorflow/python/trackable:trackable_utils_test PASSED in 7.6s //tensorflow/python/training:adadelta_test_cpu PASSED in 15.2s //tensorflow/python/training:adagrad_da_test_cpu PASSED in 8.7s //tensorflow/python/training:adagrad_test_cpu PASSED in 24.5s //tensorflow/python/training:adam_test_cpu PASSED in 19.7s //tensorflow/python/training:basic_loops_test_cpu PASSED in 8.4s //tensorflow/python/training:basic_session_run_hooks_test PASSED in 18.4s //tensorflow/python/training:checkpoint_ops_test PASSED in 7.4s //tensorflow/python/training:coordinator_test_cpu PASSED in 14.3s //tensorflow/python/training:device_setter_test_cpu PASSED in 7.9s //tensorflow/python/training:ftrl_test_cpu PASSED in 14.4s //tensorflow/python/training:gradient_descent_test_cpu PASSED in 16.8s //tensorflow/python/training:input_test PASSED in 23.8s //tensorflow/python/training:momentum_test_cpu PASSED in 13.6s //tensorflow/python/training:monitored_session_test PASSED in 38.1s //tensorflow/python/training:moving_averages_test_cpu PASSED in 13.4s //tensorflow/python/training:optimizer_test_cpu PASSED in 21.4s //tensorflow/python/training:proximal_adagrad_test_cpu PASSED in 22.2s //tensorflow/python/training:proximal_gradient_descent_test_cpu PASSED in 8.3s //tensorflow/python/training:quantize_training_test_cpu PASSED in 7.2s //tensorflow/python/training:queue_runner_test_cpu PASSED in 6.6s //tensorflow/python/training:rmsprop_test_cpu PASSED in 24.7s //tensorflow/python/training:saver_large_partitioned_variable_test PASSED in 13.7s //tensorflow/python/training:saver_test_2gpu PASSED in 31.6s //tensorflow/python/training:saver_test_cpu PASSED in 65.5s //tensorflow/python/training:server_lib_multiple_containers_test PASSED in 24.9s //tensorflow/python/training:server_lib_same_variables_clear_container_test PASSED in 9.4s //tensorflow/python/training:server_lib_same_variables_clear_test PASSED in 6.4s //tensorflow/python/training:server_lib_same_variables_no_clear_test PASSED in 7.8s //tensorflow/python/training:server_lib_sparse_job_test PASSED in 8.2s //tensorflow/python/training:server_lib_test PASSED in 17.3s //tensorflow/python/training:session_manager_test_cpu PASSED in 75.6s //tensorflow/python/training:slot_creator_test_cpu PASSED in 9.8s //tensorflow/python/training:supervisor_test PASSED in 14.8s //tensorflow/python/training:training_ops_mlir_test_cpu PASSED in 7.8s //tensorflow/python/training:training_ops_test_cpu PASSED in 8.6s //tensorflow/python/training:training_util_test PASSED in 8.9s //tensorflow/python/training:warm_starting_util_test PASSED in 24.3s //tensorflow/python/training/experimental:loss_scale_optimizer_test PASSED in 14.8s //tensorflow/python/training/experimental:loss_scale_test PASSED in 35.6s //tensorflow/python/training/experimental:mixed_precision_test_cpu PASSED in 8.4s //tensorflow/python/training/saving:saveable_object_util_test PASSED in 7.7s //tensorflow/python/util:compat_test PASSED in 6.9s //tensorflow/python/util:decorator_utils_test PASSED in 8.2s //tensorflow/python/util:deprecation_test PASSED in 13.0s //tensorflow/python/util:dispatch_test PASSED in 8.7s //tensorflow/python/util:example_parser_configuration_test PASSED in 8.0s //tensorflow/python/util:fast_module_type_test PASSED in 7.2s //tensorflow/python/util:function_parameter_canonicalizer_test PASSED in 7.7s //tensorflow/python/util:function_utils_test PASSED in 7.5s //tensorflow/python/util:keyword_args_test PASSED in 7.8s //tensorflow/python/util:lazy_loader_test PASSED in 8.2s //tensorflow/python/util:lock_util_test PASSED in 8.9s //tensorflow/python/util:module_wrapper_test PASSED in 8.0s //tensorflow/python/util:nest_test PASSED in 21.9s //tensorflow/python/util:object_identity_test PASSED in 7.1s //tensorflow/python/util:pywrap_xla_ops_test PASSED in 2.1s //tensorflow/python/util:serialization_test PASSED in 7.4s //tensorflow/python/util:tf_contextlib_test PASSED in 6.6s //tensorflow/python/util:tf_decorator_test PASSED in 16.6s //tensorflow/python/util:tf_export_test PASSED in 7.7s //tensorflow/python/util:tf_inspect_test PASSED in 7.3s //tensorflow/python/util:tf_should_use_test PASSED in 25.7s //tensorflow/python/util:tf_stack_test PASSED in 6.8s //tensorflow/python/util:traceback_utils_test PASSED in 8.6s //tensorflow/python/util:type_annotations_test PASSED in 19.3s //tensorflow/python/util:variable_utils_test PASSED in 8.2s //tensorflow/python/util:vlog_test PASSED in 7.8s //tensorflow/tools/api/tests:module_test PASSED in 17.5s //tensorflow/tools/benchmark:benchmark_model_test PASSED in 2.2s //tensorflow/tools/common:public_api_test PASSED in 1.9s //tensorflow/tools/common:traverse_test PASSED in 2.0s //tensorflow/tools/compatibility:all_renames_v2_test PASSED in 6.3s //tensorflow/tools/compatibility:ast_edits_test PASSED in 7.0s //tensorflow/tools/compatibility:test_file_v1_0 PASSED in 14.1s //tensorflow/tools/compatibility:test_file_v2_0 PASSED in 16.3s //tensorflow/tools/compatibility:tf_upgrade_test PASSED in 6.8s //tensorflow/tools/compatibility:tf_upgrade_v2_safety_test PASSED in 6.5s //tensorflow/tools/docs:tf_doctest_test PASSED in 1.3s //tensorflow/tools/graph_transforms:file_utils_test PASSED in 1.2s //tensorflow/tools/graph_transforms:transform_graph_test PASSED in 2.6s //tensorflow/tools/graph_transforms:transform_utils_test PASSED in 1.9s //tensorflow/tools/graph_transforms:transforms_test PASSED in 5.2s //tensorflow/tools/proto_text:gen_proto_text_functions_lib_test PASSED in 0.2s //tensorflow/tools/tensorflow_builder/compat_checker:compat_checker_test PASSED in 0.4s //tensorflow/tsl/c:tsl_status_helper_test PASSED in 0.1s //tensorflow/tsl/c:tsl_status_test PASSED in 0.3s //tensorflow/tsl/concurrency:async_value_ref_test PASSED in 0.7s //tensorflow/tsl/concurrency:async_value_test PASSED in 0.2s //tensorflow/tsl/concurrency:concurrent_vector_test PASSED in 0.1s //tensorflow/tsl/cuda:cudnn_version_test PASSED in 0.1s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_agent_test PASSED in 13.2s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_error_util_test PASSED in 0.8s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_recoverable_job_test PASSED in 0.4s //tensorflow/tsl/distributed_runtime/preemption:preemption_notifier_test PASSED in 7.4s //tensorflow/tsl/distributed_runtime/preemption:preemption_sync_manager_test PASSED in 5.9s //tensorflow/tsl/distributed_runtime/rpc:grpc_channel_test PASSED in 0.3s //tensorflow/tsl/distributed_runtime/rpc:grpc_util_test PASSED in 0.1s //tensorflow/tsl/framework:cancellation_test PASSED in 1.5s //tensorflow/tsl/framework:device_id_utils_test PASSED in 4.4s //tensorflow/tsl/framework/convolution:spatial_convolutions_test PASSED in 0.2s //tensorflow/tsl/lib/gtl:tsl_lib_gtl_tests PASSED in 0.1s //tensorflow/tsl/lib/hash:crc32c_test PASSED in 0.1s //tensorflow/tsl/lib/histogram:histogram_test PASSED in 0.4s //tensorflow/tsl/lib/io:buffered_inputstream_test PASSED in 0.6s //tensorflow/tsl/lib/io:cache_test PASSED in 0.2s //tensorflow/tsl/lib/io:inputbuffer_test PASSED in 1.5s //tensorflow/tsl/lib/io:inputstream_interface_test PASSED in 0.3s //tensorflow/tsl/lib/io:random_inputstream_test PASSED in 0.2s //tensorflow/tsl/lib/io:record_reader_writer_test PASSED in 2.1s //tensorflow/tsl/lib/io:recordio_test PASSED in 0.4s //tensorflow/tsl/lib/io:table_test PASSED in 5.3s //tensorflow/tsl/lib/io:zlib_buffers_test PASSED in 10.1s //tensorflow/tsl/lib/io/snappy:snappy_test PASSED in 1.2s //tensorflow/tsl/lib/math:math_util_test PASSED in 0.2s //tensorflow/tsl/lib/random:distribution_sampler_test PASSED in 0.1s //tensorflow/tsl/lib/random:philox_random_test PASSED in 0.1s //tensorflow/tsl/lib/random:random_distributions_test PASSED in 22.0s //tensorflow/tsl/lib/random:simple_philox_test PASSED in 0.4s //tensorflow/tsl/lib/random:weighted_picker_test PASSED in 11.9s //tensorflow/tsl/platform:ctstring_test PASSED in 0.1s //tensorflow/tsl/platform:denormal_test PASSED in 0.1s //tensorflow/tsl/platform:errors_test PASSED in 0.5s //tensorflow/tsl/platform:fingerprint_test PASSED in 0.1s //tensorflow/tsl/platform:float8_test PASSED in 0.9s //tensorflow/tsl/platform:hash_test PASSED in 0.1s //tensorflow/tsl/platform:integral_types_test PASSED in 0.1s //tensorflow/tsl/platform:intrusive_ptr_test PASSED in 0.6s //tensorflow/tsl/platform:logging_test PASSED in 21.0s //tensorflow/tsl/platform:mutex_test PASSED in 0.3s //tensorflow/tsl/platform:net_test PASSED in 0.5s //tensorflow/tsl/platform:numbers_test PASSED in 0.3s //tensorflow/tsl/platform:path_test PASSED in 0.1s //tensorflow/tsl/platform:port_test PASSED in 8.4s //tensorflow/tsl/platform:random_test PASSED in 1.7s //tensorflow/tsl/platform:refcount_test PASSED in 3.8s //tensorflow/tsl/platform:retrying_file_system_test PASSED in 0.2s //tensorflow/tsl/platform:retrying_utils_test PASSED in 0.1s //tensorflow/tsl/platform:scanner_test PASSED in 0.2s //tensorflow/tsl/platform:setround_test PASSED in 0.1s //tensorflow/tsl/platform:stacktrace_handler_test PASSED in 1.5s //tensorflow/tsl/platform:stacktrace_test PASSED in 0.7s //tensorflow/tsl/platform:status_matchers_test PASSED in 0.1s //tensorflow/tsl/platform:status_test PASSED in 0.2s //tensorflow/tsl/platform:statusor_test PASSED in 1.8s //tensorflow/tsl/platform:str_util_test PASSED in 0.4s //tensorflow/tsl/platform:strcat_test PASSED in 0.2s //tensorflow/tsl/platform:stringpiece_test PASSED in 0.2s //tensorflow/tsl/platform:stringprintf_test PASSED in 0.1s //tensorflow/tsl/platform:subprocess_test PASSED in 0.2s //tensorflow/tsl/platform:tstring_test PASSED in 1.1s //tensorflow/tsl/platform:unbounded_work_queue_test PASSED in 1.9s //tensorflow/tsl/platform/cloud:compute_engine_metadata_client_test PASSED in 0.5s //tensorflow/tsl/platform/cloud:compute_engine_zone_provider_test PASSED in 0.1s //tensorflow/tsl/platform/cloud:curl_http_request_test PASSED in 8.3s //tensorflow/tsl/platform/cloud:expiring_lru_cache_test PASSED in 0.2s //tensorflow/tsl/platform/cloud:gcs_dns_cache_test PASSED in 0.2s //tensorflow/tsl/platform/cloud:gcs_file_system_test PASSED in 6.1s //tensorflow/tsl/platform/cloud:gcs_throttle_test PASSED in 0.2s //tensorflow/tsl/platform/cloud:google_auth_provider_test PASSED in 0.3s //tensorflow/tsl/platform/cloud:oauth_client_test PASSED in 0.2s //tensorflow/tsl/platform/cloud:ram_file_block_cache_test PASSED in 2.3s //tensorflow/tsl/platform/cloud:time_util_test PASSED in 0.1s //tensorflow/tsl/profiler/backends/cpu:traceme_recorder_test PASSED in 0.2s //tensorflow/tsl/profiler/convert:trace_container_test PASSED in 0.2s //tensorflow/tsl/profiler/convert:trace_events_to_json_test PASSED in 0.1s //tensorflow/tsl/profiler/convert:xla_op_utils_test PASSED in 0.2s //tensorflow/tsl/profiler/convert:xplane_to_trace_events_test PASSED in 0.1s //tensorflow/tsl/profiler/lib:profiler_factory_test PASSED in 0.2s //tensorflow/tsl/profiler/lib:profiler_lock_test PASSED in 0.1s //tensorflow/tsl/profiler/lib:scoped_annotation_test PASSED in 0.1s //tensorflow/tsl/profiler/lib:traceme_encode_test PASSED in 0.1s //tensorflow/tsl/profiler/rpc/client:profiler_client_test PASSED in 3.5s //tensorflow/tsl/profiler/rpc/client:remote_profiler_session_manager_test PASSED in 3.5s //tensorflow/tsl/profiler/utils:buffer_pool_test PASSED in 0.8s //tensorflow/tsl/profiler/utils:group_events_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:parse_annotation_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:preprocess_xplane_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:tf_op_utils_test PASSED in 0.6s //tensorflow/tsl/profiler/utils:timespan_test PASSED in 0.2s //tensorflow/tsl/profiler/utils:tpu_xplane_utils_test PASSED in 0.3s //tensorflow/tsl/profiler/utils:xplane_builder_test PASSED in 0.3s //tensorflow/tsl/profiler/utils:xplane_utils_test PASSED in 0.2s //tensorflow/tsl/util:device_name_utils_test PASSED in 0.1s //tensorflow/tsl/util:stats_calculator_test PASSED in 0.1s //tensorflow/compiler/tests:complex_div_test_cpu PASSED in 8.0s Stats over 2 runs: max = 8.0s, min = 6.7s, avg = 7.3s, dev = 0.7s //tensorflow/compiler/tests:complex_div_test_cpu_mlir_bridge_test PASSED in 7.0s Stats over 2 runs: max = 7.0s, min = 6.7s, avg = 6.9s, dev = 0.1s //tensorflow/compiler/xla/tests:conditional_test_cpu PASSED in 13.4s Stats over 2 runs: max = 13.4s, min = 11.7s, avg = 12.6s, dev = 0.8s //tensorflow/python:control_flow_ops_test_cpu PASSED in 25.8s Stats over 2 runs: max = 25.8s, min = 21.4s, avg = 23.6s, dev = 2.2s //tensorflow/python/data/experimental/kernel_tests/optimization:optimization_test PASSED in 21.7s Stats over 2 runs: max = 21.7s, min = 11.6s, avg = 16.7s, dev = 5.0s //tensorflow/python/data/experimental/kernel_tests/service:metadata_test PASSED in 38.5s Stats over 2 runs: max = 38.5s, min = 35.0s, avg = 36.8s, dev = 1.7s //tensorflow/python/data/kernel_tests:padded_batch_test PASSED in 25.4s Stats over 2 runs: max = 25.4s, min = 24.5s, avg = 24.9s, dev = 0.4s //tensorflow/python/data/kernel_tests:repeat_test PASSED in 39.4s Stats over 2 runs: max = 39.4s, min = 37.4s, avg = 38.4s, dev = 1.0s //tensorflow/python/data/kernel_tests:window_test PASSED in 71.8s Stats over 2 runs: max = 71.8s, min = 48.1s, avg = 59.9s, dev = 11.9s //tensorflow/python/distribute:strategy_common_test_2gpu PASSED in 26.2s Stats over 2 runs: max = 26.2s, min = 20.3s, avg = 23.3s, dev = 2.9s //tensorflow/python/distribute:strategy_common_test_cpu PASSED in 24.2s Stats over 2 runs: max = 24.2s, min = 20.2s, avg = 22.2s, dev = 2.0s //tensorflow/python/distribute:strategy_common_test_xla_2gpu PASSED in 15.3s Stats over 2 runs: max = 15.3s, min = 14.2s, avg = 14.8s, dev = 0.5s //tensorflow/python/kernel_tests/array_ops:scatter_nd_ops_test_cpu PASSED in 10.9s Stats over 2 runs: max = 10.9s, min = 10.6s, avg = 10.7s, dev = 0.1s //tensorflow/python/kernel_tests/array_ops:scatter_ops_test_cpu PASSED in 23.2s Stats over 2 runs: max = 23.2s, min = 23.0s, avg = 23.1s, dev = 0.1s //tensorflow/python/kernel_tests/control_flow:functional_ops_test_cpu PASSED in 13.9s Stats over 2 runs: max = 13.9s, min = 13.2s, avg = 13.5s, dev = 0.4s //tensorflow/python/kernel_tests/control_flow:map_fn_test_cpu PASSED in 10.7s Stats over 2 runs: max = 10.7s, min = 9.5s, avg = 10.1s, dev = 0.6s //tensorflow/python/kernel_tests/nn_ops:atrous_conv2d_test_cpu PASSED in 31.9s Stats over 2 runs: max = 31.9s, min = 20.3s, avg = 26.1s, dev = 5.8s //tensorflow/python/kernel_tests/nn_ops:bias_op_d9m_test_cpu PASSED in 69.9s Stats over 2 runs: max = 69.9s, min = 26.6s, avg = 48.3s, dev = 21.7s //tensorflow/python/kernel_tests/nn_ops:conv2d_backprop_filter_grad_test_cpu PASSED in 59.5s Stats over 2 runs: max = 59.5s, min = 6.4s, avg = 33.0s, dev = 26.6s //tensorflow/compiler/tests:spacetobatch_op_test_cpu PASSED in 9.2s Stats over 3 runs: max = 9.2s, min = 9.0s, avg = 9.1s, dev = 0.1s //tensorflow/compiler/tests:spacetobatch_op_test_cpu_mlir_bridge_test PASSED in 10.9s Stats over 3 runs: max = 10.9s, min = 9.8s, avg = 10.2s, dev = 0.5s //tensorflow/compiler/xla/tests:triangular_solve_test_cpu PASSED in 63.1s Stats over 3 runs: max = 63.1s, min = 62.3s, avg = 62.7s, dev = 0.3s //tensorflow/core/data/service:thread_safe_buffer_test PASSED in 0.3s Stats over 3 runs: max = 0.3s, min = 0.2s, avg = 0.2s, dev = 0.1s //tensorflow/python/data/experimental/kernel_tests/service:multi_process_cluster_test PASSED in 15.4s Stats over 3 runs: max = 15.4s, min = 10.9s, avg = 13.8s, dev = 2.0s //tensorflow/python/data/kernel_tests:unique_test PASSED in 14.9s Stats over 3 runs: max = 14.9s, min = 10.9s, avg = 12.5s, dev = 1.7s //tensorflow/python/kernel_tests/array_ops:gather_op_test_cpu PASSED in 47.0s Stats over 3 runs: max = 47.0s, min = 25.7s, avg = 33.4s, dev = 9.6s //tensorflow/python/kernel_tests/array_ops:weights_broadcast_test PASSED in 9.8s Stats over 3 runs: max = 9.8s, min = 9.2s, avg = 9.5s, dev = 0.2s //tensorflow/python/kernel_tests/distributions:util_test_cpu PASSED in 14.3s Stats over 3 runs: max = 14.3s, min = 9.8s, avg = 11.7s, dev = 1.9s //tensorflow/python/kernel_tests/linalg:matrix_triangular_solve_op_test_cpu PASSED in 260.1s Stats over 3 runs: max = 260.1s, min = 8.4s, avg = 92.3s, dev = 118.6s //tensorflow/python/kernel_tests/random:multinomial_op_big_test_cpu PASSED in 14.2s Stats over 3 runs: max = 14.2s, min = 10.1s, avg = 11.5s, dev = 1.9s //tensorflow/compiler/xla/tests:dynamic_ops_test_cpu PASSED in 14.4s Stats over 4 runs: max = 14.4s, min = 13.2s, avg = 13.6s, dev = 0.5s //tensorflow/core/kernels:example_parsing_ops_test PASSED in 0.7s Stats over 4 runs: max = 0.7s, min = 0.5s, avg = 0.6s, dev = 0.1s //tensorflow/python:nn_batchnorm_test_cpu PASSED in 19.0s Stats over 4 runs: max = 19.0s, min = 13.3s, avg = 15.8s, dev = 2.1s //tensorflow/python:nn_fused_batchnorm_d9m_test_cpu PASSED in 25.5s Stats over 4 runs: max = 25.5s, min = 19.9s, avg = 23.8s, dev = 2.3s //tensorflow/python/data/experimental/kernel_tests:auto_shard_dataset_test PASSED in 38.7s Stats over 4 runs: max = 38.7s, min = 22.7s, avg = 29.4s, dev = 6.0s //tensorflow/python/data/experimental/kernel_tests:map_and_batch_test PASSED in 40.7s Stats over 4 runs: max = 40.7s, min = 21.8s, avg = 27.1s, dev = 7.9s //tensorflow/python/data/experimental/kernel_tests:parse_example_dataset_test PASSED in 25.6s Stats over 4 runs: max = 25.6s, min = 12.6s, avg = 19.0s, dev = 6.4s //tensorflow/python/data/experimental/kernel_tests:rebatch_dataset_test PASSED in 30.2s Stats over 4 runs: max = 30.2s, min = 9.0s, avg = 16.7s, dev = 8.1s //tensorflow/python/data/experimental/kernel_tests:sql_dataset_test PASSED in 35.4s Stats over 4 runs: max = 35.4s, min = 29.8s, avg = 32.6s, dev = 2.6s //tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_ft_test PASSED in 8.8s Stats over 4 runs: max = 8.8s, min = 7.3s, avg = 8.0s, dev = 0.7s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_test PASSED in 44.0s Stats over 4 runs: max = 44.0s, min = 19.9s, avg = 30.7s, dev = 10.5s //tensorflow/python/data/kernel_tests:batch_test PASSED in 36.6s Stats over 4 runs: max = 36.6s, min = 29.0s, avg = 31.5s, dev = 3.1s //tensorflow/python/data/kernel_tests:fixed_length_record_dataset_test PASSED in 17.2s Stats over 4 runs: max = 17.2s, min = 7.8s, avg = 11.9s, dev = 4.0s //tensorflow/python/data/kernel_tests:from_generator_test PASSED in 39.8s Stats over 4 runs: max = 39.8s, min = 23.1s, avg = 32.1s, dev = 6.6s //tensorflow/python/data/kernel_tests:group_by_window_test PASSED in 22.1s Stats over 4 runs: max = 22.1s, min = 7.6s, avg = 13.8s, dev = 6.2s //tensorflow/python/data/kernel_tests:ragged_batch_test PASSED in 18.2s Stats over 4 runs: max = 18.2s, min = 16.5s, avg = 17.4s, dev = 0.7s //tensorflow/python/data/kernel_tests:skip_test PASSED in 23.8s Stats over 4 runs: max = 23.8s, min = 16.1s, avg = 20.4s, dev = 3.3s //tensorflow/python/data/kernel_tests:take_test PASSED in 22.6s Stats over 4 runs: max = 22.6s, min = 21.3s, avg = 21.7s, dev = 0.5s //tensorflow/python/data/kernel_tests:take_while_test PASSED in 28.9s Stats over 4 runs: max = 28.9s, min = 25.1s, avg = 27.0s, dev = 1.6s //tensorflow/python/data/kernel_tests:text_line_dataset_test PASSED in 20.6s Stats over 4 runs: max = 20.6s, min = 15.8s, avg = 18.0s, dev = 2.2s //tensorflow/python/data/kernel_tests:zip_test PASSED in 14.4s Stats over 4 runs: max = 14.4s, min = 13.0s, avg = 13.8s, dev = 0.5s //tensorflow/python/debug/lib:dumping_callback_test_cpu PASSED in 11.9s Stats over 4 runs: max = 11.9s, min = 11.4s, avg = 11.5s, dev = 0.2s //tensorflow/python/distribute:cross_device_ops_test_cpu PASSED in 30.9s Stats over 4 runs: max = 30.9s, min = 18.8s, avg = 22.6s, dev = 4.9s //tensorflow/python/distribute:strategy_gather_test_2gpu PASSED in 24.7s Stats over 4 runs: max = 24.7s, min = 15.6s, avg = 20.1s, dev = 4.3s //tensorflow/python/distribute:strategy_gather_test_cpu PASSED in 21.3s Stats over 4 runs: max = 21.3s, min = 13.2s, avg = 17.4s, dev = 3.8s //tensorflow/python/distribute:strategy_gather_test_xla_2gpu PASSED in 21.0s Stats over 4 runs: max = 21.0s, min = 8.8s, avg = 15.2s, dev = 5.2s //tensorflow/python/framework:convert_to_constants_test PASSED in 23.2s Stats over 4 runs: max = 23.2s, min = 16.0s, avg = 19.1s, dev = 2.6s //tensorflow/python/kernel_tests:collective_ops_test_2gpu PASSED in 42.7s Stats over 4 runs: max = 42.7s, min = 40.7s, avg = 41.8s, dev = 0.8s //tensorflow/python/kernel_tests:collective_ops_test_cpu PASSED in 36.1s Stats over 4 runs: max = 36.1s, min = 33.6s, avg = 35.2s, dev = 1.0s //tensorflow/python/kernel_tests/array_ops:concat_op_test_cpu PASSED in 14.6s Stats over 4 runs: max = 14.6s, min = 12.4s, avg = 13.5s, dev = 1.0s //tensorflow/python/kernel_tests/array_ops:init_ops_test_cpu PASSED in 62.3s Stats over 4 runs: max = 62.3s, min = 20.7s, avg = 40.3s, dev = 17.1s //tensorflow/python/kernel_tests/array_ops:split_op_test_cpu PASSED in 33.9s Stats over 4 runs: max = 33.9s, min = 14.0s, avg = 21.0s, dev = 8.0s //tensorflow/python/kernel_tests/linalg:einsum_op_test_cpu PASSED in 67.1s Stats over 4 runs: max = 67.1s, min = 16.0s, avg = 37.3s, dev = 20.8s //tensorflow/python/kernel_tests/linalg:linear_operator_lower_triangular_test_cpu PASSED in 33.9s Stats over 4 runs: max = 33.9s, min = 30.2s, avg = 32.0s, dev = 1.5s //tensorflow/python/kernel_tests/nn_ops:conv_ops_test_cpu PASSED in 32.4s Stats over 4 runs: max = 32.4s, min = 26.6s, avg = 28.9s, dev = 2.3s //tensorflow/python/kernel_tests/random:random_gamma_test_cpu PASSED in 84.6s Stats over 4 runs: max = 84.6s, min = 7.5s, avg = 40.5s, dev = 33.5s //tensorflow/python/kernel_tests/signal:window_ops_test_cpu PASSED in 31.7s Stats over 4 runs: max = 31.7s, min = 29.9s, avg = 30.9s, dev = 0.8s //tensorflow/python/ops/ragged:ragged_gather_op_test PASSED in 67.7s Stats over 4 runs: max = 67.7s, min = 17.0s, avg = 39.4s, dev = 18.2s //tensorflow/python/ops/ragged:ragged_getitem_test PASSED in 48.2s Stats over 4 runs: max = 48.2s, min = 42.4s, avg = 45.8s, dev = 2.2s //tensorflow/compiler/tests:async_comp_test_cpu PASSED in 7.3s Stats over 5 runs: max = 7.3s, min = 6.5s, avg = 6.9s, dev = 0.3s //tensorflow/compiler/tests:conv3d_test_cpu PASSED in 13.4s Stats over 5 runs: max = 13.4s, min = 5.5s, avg = 8.7s, dev = 3.3s //tensorflow/compiler/tests:conv3d_test_cpu_mlir_bridge_test PASSED in 14.3s Stats over 5 runs: max = 14.3s, min = 6.4s, avg = 10.0s, dev = 3.2s //tensorflow/compiler/tests:depthwise_conv_op_test_cpu PASSED in 14.8s Stats over 5 runs: max = 14.8s, min = 9.0s, avg = 11.5s, dev = 2.2s //tensorflow/compiler/tests:depthwise_conv_op_test_cpu_mlir_bridge_test PASSED in 14.8s Stats over 5 runs: max = 14.8s, min = 9.0s, avg = 11.5s, dev = 2.2s //tensorflow/compiler/tests:fused_batchnorm_test_cpu PASSED in 7.4s Stats over 5 runs: max = 7.4s, min = 6.1s, avg = 7.0s, dev = 0.5s //tensorflow/compiler/tests:fused_batchnorm_test_cpu_mlir_bridge_test PASSED in 8.3s Stats over 5 runs: max = 8.3s, min = 7.6s, avg = 7.9s, dev = 0.2s //tensorflow/compiler/tests:image_ops_jit_compile_test_cpu PASSED in 7.3s Stats over 5 runs: max = 7.3s, min = 5.5s, avg = 6.6s, dev = 0.7s //tensorflow/compiler/tests:reduce_ops_test_cpu PASSED in 10.4s Stats over 5 runs: max = 10.4s, min = 9.3s, avg = 9.8s, dev = 0.4s //tensorflow/compiler/tests:reduce_ops_test_cpu_mlir_bridge_test PASSED in 14.4s Stats over 5 runs: max = 14.4s, min = 11.8s, avg = 13.5s, dev = 1.0s //tensorflow/compiler/tests:repeat_op_test_cpu PASSED in 8.5s Stats over 5 runs: max = 8.5s, min = 6.7s, avg = 7.3s, dev = 0.6s //tensorflow/compiler/tests:repeat_op_test_cpu_mlir_bridge_test PASSED in 14.4s Stats over 5 runs: max = 14.4s, min = 11.7s, avg = 12.6s, dev = 1.0s //tensorflow/compiler/tests:special_math_test_cpu PASSED in 132.5s Stats over 5 runs: max = 132.5s, min = 16.7s, avg = 59.1s, dev = 40.1s //tensorflow/compiler/tests:special_math_test_cpu_mlir_bridge_test PASSED in 133.3s Stats over 5 runs: max = 133.3s, min = 24.0s, avg = 66.1s, dev = 38.1s //tensorflow/compiler/xla/client/lib:self_adjoint_eig_test_cpu PASSED in 30.8s Stats over 5 runs: max = 30.8s, min = 18.4s, avg = 25.8s, dev = 5.2s //tensorflow/core/grappler/optimizers:constant_folding_test PASSED in 3.5s Stats over 5 runs: max = 3.5s, min = 2.1s, avg = 2.5s, dev = 0.5s //tensorflow/dtensor/python/tests:layout_propagation_test_cpu PASSED in 9.8s Stats over 5 runs: max = 9.8s, min = 9.5s, avg = 9.7s, dev = 0.1s //tensorflow/python/distribute:mirrored_strategy_test_2gpu PASSED in 11.3s Stats over 5 runs: max = 11.3s, min = 10.2s, avg = 10.8s, dev = 0.4s //tensorflow/python/distribute:mirrored_strategy_test_cpu PASSED in 13.2s Stats over 5 runs: max = 13.2s, min = 11.0s, avg = 11.7s, dev = 0.8s //tensorflow/python/distribute:moving_averages_test_2gpu PASSED in 18.2s Stats over 5 runs: max = 18.2s, min = 14.1s, avg = 15.9s, dev = 1.7s //tensorflow/python/distribute:moving_averages_test_cpu PASSED in 17.4s Stats over 5 runs: max = 17.4s, min = 13.9s, avg = 15.4s, dev = 1.3s //tensorflow/python/distribute:vars_test_2gpu PASSED in 15.2s Stats over 5 runs: max = 15.2s, min = 13.0s, avg = 14.1s, dev = 0.7s //tensorflow/python/distribute:vars_test_cpu PASSED in 14.1s Stats over 5 runs: max = 14.1s, min = 12.5s, avg = 13.4s, dev = 0.7s //tensorflow/python/eager:device_placement_test_cpu PASSED in 8.2s Stats over 5 runs: max = 8.2s, min = 7.6s, avg = 8.0s, dev = 0.2s //tensorflow/python/eager:forwardprop_test_cpu PASSED in 112.3s Stats over 5 runs: max = 112.3s, min = 34.8s, avg = 64.5s, dev = 26.0s //tensorflow/python/eager/polymorphic_function:gradients_test_cpu PASSED in 42.0s Stats over 5 runs: max = 42.0s, min = 36.6s, avg = 39.1s, dev = 2.1s //tensorflow/python/kernel_tests/linalg:cholesky_op_test_cpu PASSED in 57.2s Stats over 5 runs: max = 57.2s, min = 33.4s, avg = 45.3s, dev = 8.7s //tensorflow/python/kernel_tests/linalg:linear_operator_adjoint_test_cpu PASSED in 38.2s Stats over 5 runs: max = 38.2s, min = 33.8s, avg = 36.0s, dev = 1.7s //tensorflow/python/kernel_tests/linalg:linear_operator_composition_test_cpu PASSED in 40.0s Stats over 5 runs: max = 40.0s, min = 34.4s, avg = 36.5s, dev = 1.9s //tensorflow/python/kernel_tests/linalg:linear_operator_diag_test_cpu PASSED in 21.2s Stats over 5 runs: max = 21.2s, min = 19.3s, avg = 20.4s, dev = 0.7s //tensorflow/python/kernel_tests/linalg:linear_operator_full_matrix_test_cpu PASSED in 26.1s Stats over 5 runs: max = 26.1s, min = 23.2s, avg = 24.7s, dev = 0.9s //tensorflow/python/kernel_tests/linalg:linear_operator_householder_test_cpu PASSED in 29.8s Stats over 5 runs: max = 29.8s, min = 25.2s, avg = 27.6s, dev = 1.8s //tensorflow/python/kernel_tests/linalg:linear_operator_identity_test_cpu PASSED in 28.5s Stats over 5 runs: max = 28.5s, min = 28.1s, avg = 28.4s, dev = 0.2s //tensorflow/python/kernel_tests/linalg:linear_operator_inversion_test_cpu PASSED in 23.9s Stats over 5 runs: max = 23.9s, min = 21.3s, avg = 22.3s, dev = 1.0s //tensorflow/python/kernel_tests/linalg:linear_operator_permutation_test_cpu PASSED in 18.0s Stats over 5 runs: max = 18.0s, min = 16.1s, avg = 17.0s, dev = 0.7s //tensorflow/python/kernel_tests/linalg:linear_operator_toeplitz_test_cpu PASSED in 12.4s Stats over 5 runs: max = 12.4s, min = 10.2s, avg = 11.4s, dev = 0.9s //tensorflow/python/kernel_tests/linalg:linear_operator_tridiag_test_cpu PASSED in 95.7s Stats over 5 runs: max = 95.7s, min = 86.5s, avg = 92.5s, dev = 3.3s //tensorflow/python/kernel_tests/linalg:linear_operator_util_test_cpu PASSED in 7.3s Stats over 5 runs: max = 7.3s, min = 6.6s, avg = 7.0s, dev = 0.3s //tensorflow/python/kernel_tests/linalg:linear_operator_zeros_test_cpu PASSED in 18.4s Stats over 5 runs: max = 18.4s, min = 15.7s, avg = 17.0s, dev = 1.1s //tensorflow/python/kernel_tests/nn_ops:fractional_avg_pool_op_test PASSED in 13.3s Stats over 5 runs: max = 13.3s, min = 7.5s, avg = 9.1s, dev = 2.2s //tensorflow/python/kernel_tests/nn_ops:fractional_max_pool_op_test PASSED in 13.8s Stats over 5 runs: max = 13.8s, min = 7.2s, avg = 8.7s, dev = 2.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_ops_test_cpu PASSED in 26.5s Stats over 5 runs: max = 26.5s, min = 7.5s, avg = 12.5s, dev = 7.1s //tensorflow/python/ops/parallel_for:math_test_cpu PASSED in 91.8s Stats over 5 runs: max = 91.8s, min = 23.9s, avg = 49.6s, dev = 24.9s //tensorflow/python/distribute:cross_device_ops_test_2gpu FLAKY, failed in 1 out of 5 in 28.7s Stats over 5 runs: max = 28.7s, min = 20.7s, avg = 24.3s, dev = 3.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/distribute/cross_device_ops_test_2gpu/shard_4_of_4/test_attempts/attempt_1.log //tensorflow/compiler/tests:scan_ops_test_cpu PASSED in 14.7s Stats over 6 runs: max = 14.7s, min = 9.5s, avg = 12.0s, dev = 1.6s //tensorflow/compiler/tests:scan_ops_test_cpu_mlir_bridge_test PASSED in 20.6s Stats over 6 runs: max = 20.6s, min = 12.4s, avg = 16.2s, dev = 2.5s //tensorflow/python:accumulate_n_benchmark_cpu PASSED in 6.9s Stats over 6 runs: max = 6.9s, min = 6.0s, avg = 6.4s, dev = 0.3s //tensorflow/python/data/experimental/kernel_tests:make_batched_features_dataset_test PASSED in 21.3s Stats over 6 runs: max = 21.3s, min = 6.2s, avg = 12.8s, dev = 6.2s //tensorflow/python/kernel_tests/array_ops:diag_op_test_cpu PASSED in 63.6s Stats over 6 runs: max = 63.6s, min = 8.2s, avg = 19.9s, dev = 19.7s //tensorflow/python/kernel_tests/math_ops:reduction_ops_test_cpu PASSED in 49.0s Stats over 6 runs: max = 49.0s, min = 22.5s, avg = 33.6s, dev = 8.2s //tensorflow/python/distribute/experimental/rpc:rpc_ops_test PASSED in 12.3s Stats over 7 runs: max = 12.3s, min = 8.6s, avg = 10.1s, dev = 1.4s //tensorflow/compiler/tests:matrix_diag_ops_test_cpu PASSED in 58.7s Stats over 8 runs: max = 58.7s, min = 6.2s, avg = 22.7s, dev = 17.8s //tensorflow/compiler/tests:matrix_diag_ops_test_cpu_mlir_bridge_test PASSED in 49.7s Stats over 8 runs: max = 49.7s, min = 6.7s, avg = 22.2s, dev = 15.3s //tensorflow/python/data/experimental/kernel_tests:csv_dataset_test PASSED in 36.5s Stats over 8 runs: max = 36.5s, min = 9.8s, avg = 19.7s, dev = 10.4s //tensorflow/python/data/experimental/kernel_tests:parallel_interleave_test PASSED in 37.2s Stats over 8 runs: max = 37.2s, min = 12.0s, avg = 25.4s, dev = 8.8s //tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_ft_test PASSED in 44.7s Stats over 8 runs: max = 44.7s, min = 14.0s, avg = 28.8s, dev = 13.1s //tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_test PASSED in 48.3s Stats over 8 runs: max = 48.3s, min = 16.1s, avg = 26.4s, dev = 10.8s //tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_test PASSED in 23.3s Stats over 8 runs: max = 23.3s, min = 6.8s, avg = 12.8s, dev = 6.3s //tensorflow/python/data/experimental/kernel_tests/service:fault_tolerance_test PASSED in 18.7s Stats over 8 runs: max = 18.7s, min = 11.3s, avg = 13.9s, dev = 2.5s //tensorflow/python/data/kernel_tests:filter_test PASSED in 20.2s Stats over 8 runs: max = 20.2s, min = 13.4s, avg = 16.3s, dev = 2.4s //tensorflow/python/data/kernel_tests:flat_map_test PASSED in 21.8s Stats over 8 runs: max = 21.8s, min = 13.8s, avg = 17.3s, dev = 3.3s //tensorflow/python/data/kernel_tests:shard_test PASSED in 19.5s Stats over 8 runs: max = 19.5s, min = 14.5s, avg = 17.1s, dev = 1.9s //tensorflow/python/data/kernel_tests:shuffle_test PASSED in 61.5s Stats over 8 runs: max = 61.5s, min = 26.8s, avg = 33.5s, dev = 10.7s //tensorflow/python/data/kernel_tests:tf_record_dataset_test PASSED in 33.1s Stats over 8 runs: max = 33.1s, min = 12.6s, avg = 23.0s, dev = 5.7s //tensorflow/python/distribute/failure_handling:failure_handler_test PASSED in 47.8s Stats over 8 runs: max = 47.8s, min = 21.4s, avg = 34.6s, dev = 8.5s //tensorflow/python/distribute/failure_handling:gce_failure_handler_test PASSED in 93.3s Stats over 8 runs: max = 93.3s, min = 19.9s, avg = 44.8s, dev = 27.6s //tensorflow/python/kernel_tests/linalg:linalg_ops_test_cpu PASSED in 45.6s Stats over 8 runs: max = 45.6s, min = 26.7s, avg = 38.3s, dev = 6.3s //tensorflow/python/kernel_tests/linalg:linear_operator_block_diag_test_cpu PASSED in 60.5s Stats over 8 runs: max = 60.5s, min = 42.3s, avg = 51.4s, dev = 6.7s //tensorflow/python/kernel_tests/linalg:linear_operator_block_lower_triangular_test_cpu PASSED in 52.1s Stats over 8 runs: max = 52.1s, min = 34.9s, avg = 44.1s, dev = 6.2s //tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_d9m_test_cpu PASSED in 66.0s Stats over 8 runs: max = 66.0s, min = 7.1s, avg = 16.9s, dev = 19.6s //tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_test_cpu PASSED in 8.1s Stats over 8 runs: max = 8.1s, min = 7.2s, avg = 7.6s, dev = 0.3s //tensorflow/python/kernel_tests/signal:fft_ops_test_cpu PASSED in 22.1s Stats over 8 runs: max = 22.1s, min = 7.8s, avg = 12.9s, dev = 5.2s //tensorflow/python/ops/ragged:dynamic_ragged_shape_test PASSED in 46.4s Stats over 8 runs: max = 46.4s, min = 29.6s, avg = 37.4s, dev = 5.7s //tensorflow/python/ops/ragged:ragged_tensor_test PASSED in 23.2s Stats over 8 runs: max = 23.2s, min = 11.7s, avg = 14.8s, dev = 3.4s //tensorflow/compiler/tests:bincount_op_test_cpu PASSED in 6.6s Stats over 10 runs: max = 6.6s, min = 4.5s, avg = 5.5s, dev = 0.5s //tensorflow/compiler/tests:conv2d_test_cpu PASSED in 6.8s Stats over 10 runs: max = 6.8s, min = 3.4s, avg = 4.9s, dev = 1.2s //tensorflow/compiler/tests:conv2d_test_cpu_mlir_bridge_test PASSED in 7.4s Stats over 10 runs: max = 7.4s, min = 5.9s, avg = 6.9s, dev = 0.6s //tensorflow/compiler/tests:random_ops_test_cpu PASSED in 21.0s Stats over 10 runs: max = 21.0s, min = 12.1s, avg = 16.3s, dev = 2.7s //tensorflow/compiler/tests:random_ops_test_cpu_mlir_bridge_test PASSED in 21.2s Stats over 10 runs: max = 21.2s, min = 14.3s, avg = 17.6s, dev = 2.0s //tensorflow/compiler/tests:stateless_random_ops_test_cpu PASSED in 70.8s Stats over 10 runs: max = 70.8s, min = 35.5s, avg = 51.2s, dev = 11.5s //tensorflow/compiler/tests:stateless_random_ops_test_cpu_mlir_bridge_test PASSED in 70.3s Stats over 10 runs: max = 70.3s, min = 35.7s, avg = 52.1s, dev = 11.1s //tensorflow/compiler/xla/client/lib:svd_test_cpu PASSED in 69.6s Stats over 10 runs: max = 69.6s, min = 10.5s, avg = 26.7s, dev = 22.8s //tensorflow/compiler/xla/client/lib:tridiagonal_test_cpu PASSED in 10.7s Stats over 10 runs: max = 10.7s, min = 9.4s, avg = 10.0s, dev = 0.4s //tensorflow/compiler/xla/service/cpu:cpu_runtime_test PASSED in 13.4s Stats over 10 runs: max = 13.4s, min = 1.0s, avg = 9.2s, dev = 4.2s //tensorflow/python:special_math_ops_test_cpu PASSED in 45.6s Stats over 10 runs: max = 45.6s, min = 6.0s, avg = 12.1s, dev = 11.4s //tensorflow/python/data/kernel_tests:rejection_resample_test PASSED in 19.8s Stats over 10 runs: max = 19.8s, min = 6.3s, avg = 11.1s, dev = 3.8s //tensorflow/python/distribute:input_lib_test_2gpu PASSED in 35.4s Stats over 10 runs: max = 35.4s, min = 21.2s, avg = 26.1s, dev = 4.0s //tensorflow/python/distribute:input_lib_type_spec_test_2gpu PASSED in 17.3s Stats over 10 runs: max = 17.3s, min = 7.0s, avg = 12.2s, dev = 3.4s //tensorflow/python/distribute:input_lib_type_spec_test_cpu PASSED in 16.2s Stats over 10 runs: max = 16.2s, min = 6.0s, avg = 11.8s, dev = 3.7s //tensorflow/python/framework:config_vgpu_test_2gpu PASSED in 6.8s Stats over 10 runs: max = 6.8s, min = 5.3s, avg = 6.2s, dev = 0.5s //tensorflow/python/framework:config_vgpu_test_cpu PASSED in 20.3s Stats over 10 runs: max = 20.3s, min = 16.6s, avg = 17.7s, dev = 1.0s //tensorflow/python/framework:function_test_cpu PASSED in 57.5s Stats over 10 runs: max = 57.5s, min = 4.9s, avg = 12.5s, dev = 15.3s //tensorflow/python/grappler:cluster_test_cpu PASSED in 8.5s Stats over 10 runs: max = 8.5s, min = 4.5s, avg = 6.6s, dev = 1.3s //tensorflow/python/kernel_tests/array_ops:array_ops_test_cpu PASSED in 16.3s Stats over 10 runs: max = 16.3s, min = 10.0s, avg = 12.7s, dev = 2.1s //tensorflow/python/kernel_tests/array_ops:inplace_ops_test_cpu PASSED in 6.8s Stats over 10 runs: max = 6.8s, min = 3.1s, avg = 5.6s, dev = 1.3s //tensorflow/python/kernel_tests/data_structures:tensor_array_ops_test_cpu PASSED in 9.6s Stats over 10 runs: max = 9.6s, min = 7.2s, avg = 8.3s, dev = 0.9s //tensorflow/python/kernel_tests/linalg:linear_operator_kronecker_test_cpu PASSED in 31.2s Stats over 10 runs: max = 31.2s, min = 26.0s, avg = 28.2s, dev = 1.9s //tensorflow/python/kernel_tests/linalg:linear_operator_low_rank_update_test_cpu PASSED in 70.5s Stats over 10 runs: max = 70.5s, min = 62.5s, avg = 66.9s, dev = 2.3s //tensorflow/python/kernel_tests/linalg:tridiagonal_matmul_op_test_cpu PASSED in 90.8s Stats over 10 runs: max = 90.8s, min = 6.5s, avg = 15.4s, dev = 25.2s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_ops_test_cpu PASSED in 83.2s Stats over 10 runs: max = 83.2s, min = 44.7s, avg = 63.7s, dev = 11.9s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_test_cpu PASSED in 25.7s Stats over 10 runs: max = 25.7s, min = 6.7s, avg = 14.6s, dev = 7.3s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_test_cpu PASSED in 36.4s Stats over 10 runs: max = 36.4s, min = 15.1s, avg = 20.8s, dev = 7.2s //tensorflow/python/kernel_tests/nn_ops:rnn_test_cpu PASSED in 9.9s Stats over 10 runs: max = 9.9s, min = 5.3s, avg = 7.8s, dev = 1.6s //tensorflow/python/kernel_tests/random:random_index_shuffle_test PASSED in 15.9s Stats over 10 runs: max = 15.9s, min = 7.5s, avg = 10.5s, dev = 2.0s //tensorflow/python/kernel_tests/random:stateless_random_ops_test_cpu PASSED in 113.7s Stats over 10 runs: max = 113.7s, min = 18.5s, avg = 65.3s, dev = 44.6s //tensorflow/python/ops/ragged:ragged_tensor_supported_values_test PASSED in 17.1s Stats over 10 runs: max = 17.1s, min = 14.5s, avg = 15.8s, dev = 0.7s //tensorflow/python/saved_model:load_test_cpu PASSED in 43.3s Stats over 10 runs: max = 43.3s, min = 23.2s, avg = 28.5s, dev = 5.2s //tensorflow/dtensor/python/tests:input_util_test FLAKY, failed in 2 out of 10 in 26.8s Stats over 10 runs: max = 26.8s, min = 14.6s, avg = 21.1s, dev = 3.9s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/dtensor/python/tests/input_util_test/shard_1_of_8/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/dtensor/python/tests/input_util_test/shard_1_of_8/test_attempts/attempt_2.log //tensorflow/python/distribute:input_lib_test_cpu FLAKY, failed in 1 out of 11 in 33.6s Stats over 11 runs: max = 33.6s, min = 23.4s, avg = 26.7s, dev = 2.9s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/distribute/input_lib_test_cpu/shard_1_of_10/test_attempts/attempt_1.log //tensorflow/compiler/tests:fft_test_cpu PASSED in 24.5s Stats over 12 runs: max = 24.5s, min = 7.4s, avg = 14.3s, dev = 5.9s //tensorflow/compiler/xla/service:triangular_solve_expander_test PASSED in 5.6s Stats over 12 runs: max = 5.6s, min = 3.0s, avg = 4.2s, dev = 0.7s //tensorflow/python/data/experimental/kernel_tests:group_by_reducer_test PASSED in 24.7s Stats over 12 runs: max = 24.7s, min = 6.1s, avg = 12.0s, dev = 6.0s //tensorflow/python/data/kernel_tests:choose_from_datasets_test PASSED in 11.4s Stats over 12 runs: max = 11.4s, min = 5.0s, avg = 7.9s, dev = 2.2s //tensorflow/python/data/kernel_tests:memory_cleanup_test_cpu PASSED in 10.2s Stats over 12 runs: max = 10.2s, min = 6.4s, avg = 8.0s, dev = 1.4s //tensorflow/python/distribute:multi_process_runner_test_2gpu PASSED in 222.1s Stats over 12 runs: max = 222.1s, min = 11.1s, avg = 48.7s, dev = 58.2s //tensorflow/python/distribute:multi_process_runner_test_cpu PASSED in 221.6s Stats over 12 runs: max = 221.6s, min = 11.1s, avg = 48.9s, dev = 58.0s //tensorflow/python/eager/polymorphic_function:polymorphic_function_test_cpu PASSED in 20.2s Stats over 15 runs: max = 20.2s, min = 8.4s, avg = 12.8s, dev = 4.0s //tensorflow/python/kernel_tests/linalg:linear_operator_circulant_test_cpu PASSED in 76.2s Stats over 15 runs: max = 76.2s, min = 59.2s, avg = 67.7s, dev = 4.2s //tensorflow/python/kernel_tests/nn_ops:rnn_cell_test_cpu PASSED in 62.1s Stats over 15 runs: max = 62.1s, min = 28.9s, avg = 35.1s, dev = 8.1s //tensorflow/compiler/tests:ftrl_test_cpu PASSED in 10.1s Stats over 16 runs: max = 10.1s, min = 5.5s, avg = 7.4s, dev = 1.7s //tensorflow/compiler/tests:ternary_ops_test_cpu PASSED in 10.5s Stats over 16 runs: max = 10.5s, min = 4.0s, avg = 7.1s, dev = 1.9s //tensorflow/compiler/tests:ternary_ops_test_cpu_mlir_bridge_test PASSED in 12.7s Stats over 16 runs: max = 12.7s, min = 4.5s, avg = 7.3s, dev = 2.2s //tensorflow/python:image_ops_test_cpu PASSED in 17.8s Stats over 16 runs: max = 17.8s, min = 9.0s, avg = 12.4s, dev = 2.3s //tensorflow/python/data/experimental/kernel_tests/service:dynamic_sharding_test PASSED in 24.8s Stats over 16 runs: max = 24.8s, min = 13.6s, avg = 18.8s, dev = 3.8s //tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test PASSED in 24.7s Stats over 16 runs: max = 24.7s, min = 6.7s, avg = 13.9s, dev = 5.4s //tensorflow/python/data/kernel_tests:snapshot_test PASSED in 26.9s Stats over 16 runs: max = 26.9s, min = 8.0s, avg = 16.6s, dev = 4.8s //tensorflow/python/kernel_tests/control_flow:control_flow_ops_py_test_cpu PASSED in 27.9s Stats over 16 runs: max = 27.9s, min = 6.9s, avg = 10.3s, dev = 4.7s //tensorflow/python/kernel_tests/linalg:matrix_exponential_op_test PASSED in 31.7s Stats over 16 runs: max = 31.7s, min = 20.5s, avg = 25.4s, dev = 3.9s //tensorflow/python/kernel_tests/signal:dct_ops_test_cpu PASSED in 11.0s Stats over 16 runs: max = 11.0s, min = 6.9s, avg = 8.8s, dev = 1.4s //tensorflow/python/ops/parallel_for:control_flow_ops_test_cpu PASSED in 59.5s Stats over 16 runs: max = 59.5s, min = 9.9s, avg = 20.1s, dev = 11.2s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test PASSED in 10.3s Stats over 17 runs: max = 10.3s, min = 4.2s, avg = 5.7s, dev = 1.4s //tensorflow/python/data/kernel_tests:map_test PASSED in 44.1s Stats over 19 runs: max = 44.1s, min = 13.7s, avg = 20.4s, dev = 7.2s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu PASSED in 7.2s Stats over 20 runs: max = 7.2s, min = 5.3s, avg = 6.2s, dev = 0.6s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu_mlir_bridge_test PASSED in 7.7s Stats over 20 runs: max = 7.7s, min = 3.3s, avg = 5.3s, dev = 1.4s //tensorflow/compiler/tests:pooling_ops_test_cpu PASSED in 12.2s Stats over 20 runs: max = 12.2s, min = 4.0s, avg = 6.3s, dev = 1.7s //tensorflow/compiler/tests:pooling_ops_test_cpu_mlir_bridge_test PASSED in 12.0s Stats over 20 runs: max = 12.0s, min = 3.5s, avg = 6.7s, dev = 2.1s //tensorflow/compiler/xla/tests:convolution_dimension_numbers_test_cpu PASSED in 12.5s Stats over 20 runs: max = 12.5s, min = 10.1s, avg = 11.3s, dev = 0.7s //tensorflow/compiler/xla/tests:dot_operation_single_threaded_runtime_test_cpu PASSED in 17.8s Stats over 20 runs: max = 17.8s, min = 14.2s, avg = 16.2s, dev = 0.9s //tensorflow/compiler/xla/tests:dot_operation_test_cpu PASSED in 19.1s Stats over 20 runs: max = 19.1s, min = 14.2s, avg = 16.4s, dev = 1.4s //tensorflow/compiler/xla/tests:prng_test_cpu PASSED in 10.8s Stats over 20 runs: max = 10.8s, min = 9.4s, avg = 10.1s, dev = 0.3s //tensorflow/compiler/xla/tests:reduce_window_test_cpu PASSED in 42.8s Stats over 20 runs: max = 42.8s, min = 9.8s, avg = 19.9s, dev = 10.1s //tensorflow/python/autograph/tests:loop_control_flow_test PASSED in 19.5s Stats over 20 runs: max = 19.5s, min = 12.6s, avg = 15.9s, dev = 2.1s //tensorflow/python/kernel_tests:metrics_test PASSED in 42.5s Stats over 20 runs: max = 42.5s, min = 7.9s, avg = 18.8s, dev = 8.9s //tensorflow/python/kernel_tests/array_ops:matrix_band_part_op_test_cpu PASSED in 9.9s Stats over 20 runs: max = 9.9s, min = 3.1s, avg = 5.2s, dev = 2.3s //tensorflow/python/kernel_tests/data_structures:barrier_ops_test PASSED in 12.3s Stats over 20 runs: max = 12.3s, min = 3.6s, avg = 6.2s, dev = 2.2s //tensorflow/python/kernel_tests/linalg:eig_op_test PASSED in 45.9s Stats over 20 runs: max = 45.9s, min = 3.8s, avg = 15.3s, dev = 14.0s //tensorflow/python/kernel_tests/linalg:linalg_grad_test_cpu PASSED in 142.8s Stats over 20 runs: max = 142.8s, min = 56.1s, avg = 85.4s, dev = 23.3s //tensorflow/python/kernel_tests/linalg:norm_op_test_cpu PASSED in 23.9s Stats over 20 runs: max = 23.9s, min = 3.2s, avg = 15.4s, dev = 7.0s //tensorflow/python/kernel_tests/linalg:normalize_op_test_cpu PASSED in 19.7s Stats over 20 runs: max = 19.7s, min = 6.2s, avg = 13.8s, dev = 3.5s //tensorflow/python/kernel_tests/linalg:qr_op_test_cpu PASSED in 167.1s Stats over 20 runs: max = 167.1s, min = 30.5s, avg = 85.7s, dev = 40.8s //tensorflow/python/kernel_tests/linalg:self_adjoint_eig_op_test_cpu PASSED in 25.0s Stats over 20 runs: max = 25.0s, min = 8.0s, avg = 13.3s, dev = 5.7s //tensorflow/python/kernel_tests/math_ops:batch_matmul_op_test_cpu PASSED in 23.5s Stats over 20 runs: max = 23.5s, min = 4.8s, avg = 12.6s, dev = 6.6s //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu PASSED in 17.1s Stats over 20 runs: max = 17.1s, min = 11.4s, avg = 13.8s, dev = 1.7s //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu PASSED in 55.3s Stats over 20 runs: max = 55.3s, min = 4.3s, avg = 23.7s, dev = 17.2s //tensorflow/python/kernel_tests/nn_ops:embedding_ops_test_cpu PASSED in 34.4s Stats over 20 runs: max = 34.4s, min = 11.9s, avg = 16.9s, dev = 4.8s //tensorflow/python/data/experimental/kernel_tests/service:local_workers_test PASSED in 23.8s Stats over 24 runs: max = 23.8s, min = 8.4s, avg = 17.3s, dev = 3.5s //tensorflow/python/data/kernel_tests:interleave_test PASSED in 21.6s Stats over 24 runs: max = 21.6s, min = 6.9s, avg = 13.6s, dev = 4.3s //tensorflow/python/data/kernel_tests:sample_from_datasets_test PASSED in 18.8s Stats over 24 runs: max = 18.8s, min = 4.3s, avg = 9.7s, dev = 3.9s //tensorflow/compiler/xla/tests:array_elementwise_ops_test_cpu PASSED in 12.5s Stats over 25 runs: max = 12.5s, min = 9.8s, avg = 10.8s, dev = 0.7s //tensorflow/compiler/xla/tests:select_and_scatter_test_cpu PASSED in 43.6s Stats over 25 runs: max = 43.6s, min = 10.1s, avg = 15.1s, dev = 8.3s //tensorflow/compiler/xla/tests:convolution_variants_test_cpu PASSED in 17.6s Stats over 30 runs: max = 17.6s, min = 11.8s, avg = 12.4s, dev = 1.0s //tensorflow/compiler/xla/tests:iota_test_cpu PASSED in 20.2s Stats over 30 runs: max = 20.2s, min = 16.7s, avg = 18.4s, dev = 1.0s //tensorflow/compiler/xla/tests:params_test_cpu PASSED in 14.1s Stats over 30 runs: max = 14.1s, min = 11.9s, avg = 13.0s, dev = 0.6s //tensorflow/compiler/xla/tests:reshape_test_cpu PASSED in 13.1s Stats over 30 runs: max = 13.1s, min = 10.2s, avg = 11.6s, dev = 0.8s //tensorflow/python/kernel_tests/nn_ops:conv_ops_3d_test_cpu PASSED in 17.1s Stats over 30 runs: max = 17.1s, min = 2.8s, avg = 6.9s, dev = 3.2s //tensorflow/compiler/xla/tests:reduce_test_cpu PASSED in 14.4s Stats over 31 runs: max = 14.4s, min = 12.0s, avg = 13.4s, dev = 0.7s //tensorflow/compiler/tests:unary_ops_test_cpu PASSED in 19.1s Stats over 32 runs: max = 19.1s, min = 2.7s, avg = 7.0s, dev = 3.6s //tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test PASSED in 17.3s Stats over 32 runs: max = 17.3s, min = 3.0s, avg = 7.3s, dev = 3.8s //tensorflow/compiler/xla/tests:scalar_computations_test_cpu PASSED in 14.5s Stats over 32 runs: max = 14.5s, min = 9.8s, avg = 11.4s, dev = 1.0s //tensorflow/python/data/experimental/kernel_tests/service:auto_shard_test PASSED in 18.2s Stats over 32 runs: max = 18.2s, min = 3.5s, avg = 11.0s, dev = 3.8s //tensorflow/python/data/experimental/kernel_tests/service:data_service_ops_test PASSED in 34.2s Stats over 32 runs: max = 34.2s, min = 10.4s, avg = 22.7s, dev = 6.2s //tensorflow/compiler/xla/tests:batch_normalization_test_cpu PASSED in 19.0s Stats over 40 runs: max = 19.0s, min = 11.6s, avg = 14.2s, dev = 1.7s //tensorflow/compiler/xla/tests:bfloat16_test_cpu PASSED in 16.0s Stats over 40 runs: max = 16.0s, min = 12.6s, avg = 14.5s, dev = 0.9s //tensorflow/compiler/xla/tests:conv_depthwise_backprop_filter_test_cpu PASSED in 15.9s Stats over 40 runs: max = 15.9s, min = 12.4s, avg = 14.5s, dev = 0.6s //tensorflow/compiler/xla/tests:slice_test_cpu PASSED in 15.3s Stats over 40 runs: max = 15.3s, min = 11.1s, avg = 13.2s, dev = 1.1s //tensorflow/core/kernels:stochastic_cast_op_test PASSED in 1.6s Stats over 48 runs: max = 1.6s, min = 0.4s, avg = 0.6s, dev = 0.3s //tensorflow/compiler/mlir/quantization/tensorflow/python:quantize_model_test PASSED in 49.1s Stats over 50 runs: max = 49.1s, min = 24.3s, avg = 33.5s, dev = 7.0s //tensorflow/compiler/tests:sort_ops_test_cpu PASSED in 44.4s Stats over 50 runs: max = 44.4s, min = 3.1s, avg = 13.0s, dev = 8.9s //tensorflow/compiler/tests:sort_ops_test_cpu_mlir_bridge_test PASSED in 37.1s Stats over 50 runs: max = 37.1s, min = 2.7s, avg = 9.6s, dev = 7.8s //tensorflow/compiler/xla/tests:conv_depthwise_test_cpu PASSED in 22.6s Stats over 50 runs: max = 22.6s, min = 9.0s, avg = 12.6s, dev = 2.9s //tensorflow/compiler/xla/tests:convolution_test_1d_no_vmodule_cpu PASSED in 19.5s Stats over 50 runs: max = 19.5s, min = 13.4s, avg = 16.7s, dev = 1.4s //tensorflow/compiler/xla/tests:convolution_test_cpu PASSED in 23.0s Stats over 50 runs: max = 23.0s, min = 15.0s, avg = 18.4s, dev = 1.5s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_grad_test_cpu PASSED in 14.2s Stats over 50 runs: max = 14.2s, min = 4.6s, avg = 8.3s, dev = 2.6s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_grad_test_cpu PASSED in 7.5s Stats over 50 runs: max = 7.5s, min = 3.1s, avg = 4.2s, dev = 1.1s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_sparse_mat_mul_grad_test_cpu PASSED in 7.5s Stats over 50 runs: max = 7.5s, min = 2.9s, avg = 4.5s, dev = 1.3s //tensorflow/python/kernel_tests/math_ops:cwise_ops_binary_test_cpu PASSED in 30.7s Stats over 50 runs: max = 30.7s, min = 7.6s, avg = 17.0s, dev = 6.2s //tensorflow/python/kernel_tests/math_ops:cwise_ops_test_cpu PASSED in 7.7s Stats over 50 runs: max = 7.7s, min = 3.0s, avg = 4.2s, dev = 1.2s //tensorflow/python/kernel_tests/math_ops:cwise_ops_unary_test_cpu PASSED in 12.0s Stats over 50 runs: max = 12.0s, min = 3.1s, avg = 4.7s, dev = 2.0s Executed 3791 out of 3791 tests: 3791 tests pass. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.