==================== Test output for //tensorflow/c/eager:c_api_test_cpu: 2023-04-21 03:35:06.024172: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2023-04-21 03:35:06.024376: E tensorflow/tsl/lib/monitoring/collection_registry.cc:81] Cannot register 2 metrics with the same name: /tensorflow/core/bfc_allocator_delay [==========] Running 48 tests from 2 test suites. [----------] Global test environment set-up. [----------] 2 tests from CApiDebug [ RUN ] CApiDebug.ScalarCPU [ OK ] CApiDebug.ScalarCPU (42 ms) [ RUN ] CApiDebug.2DCPU [ OK ] CApiDebug.2DCPU (9 ms) [----------] 2 tests from CApiDebug (52 ms total) [----------] 46 tests from CAPI [ RUN ] CAPI.Context [ OK ] CAPI.Context (9 ms) [ RUN ] CAPI.TensorHandle [ OK ] CAPI.TensorHandle (10 ms) [ RUN ] CAPI.TensorHandleCopyBetweenDevices [ OK ] CAPI.TensorHandleCopyBetweenDevices (77 ms) [ RUN ] CAPI.TensorHandleCopyBetweenDevicesAsync [ OK ] CAPI.TensorHandleCopyBetweenDevicesAsync (89 ms) [ RUN ] CAPI.TensorHandleCopyBetweenDevicesError [ OK ] CAPI.TensorHandleCopyBetweenDevicesError (40 ms) [ RUN ] CAPI.TensorHandleCopyBetweenDevicesErrorAsync [ OK ] CAPI.TensorHandleCopyBetweenDevicesErrorAsync (6 ms) [ RUN ] CAPI.TensorHandleCopyBetweenTwoGPUDevices [ OK ] CAPI.TensorHandleCopyBetweenTwoGPUDevices (7 ms) [ RUN ] CAPI.TensorHandleCopyBetweenTwoGPUDevicesAsync [ OK ] CAPI.TensorHandleCopyBetweenTwoGPUDevicesAsync (35 ms) [ RUN ] CAPI.TensorHandleSilentCopy [ OK ] CAPI.TensorHandleSilentCopy (14 ms) [ RUN ] CAPI.TensorHandleSilentCopyAsync [ OK ] CAPI.TensorHandleSilentCopyAsync (27 ms) [ RUN ] CAPI.TensorHandleSilentCopyLocalPolicy [ OK ] CAPI.TensorHandleSilentCopyLocalPolicy (7 ms) [ RUN ] CAPI.TensorHandleSilentCopyLocalPolicyAsync [ OK ] CAPI.TensorHandleSilentCopyLocalPolicyAsync (14 ms) [ RUN ] CAPI.TensorHandleNullptr [ OK ] CAPI.TensorHandleNullptr (0 ms) [ RUN ] CAPI.TensorHandleDevices [ OK ] CAPI.TensorHandleDevices (10 ms) [ RUN ] CAPI.ExecuteAdd 2023-04-21 03:35:06.512864: I tensorflow/c/eager/c_api_test_util.cc:416] Found CPU device /job:localhost/replica:0/task:0/device:CPU:0 [ OK ] CAPI.ExecuteAdd (50 ms) [ DISABLED ] CAPI.DISABLED_ExecuteAddAsync [ RUN ] CAPI.ExecuteAddForward 2023-04-21 03:35:06.537500: I tensorflow/c/eager/c_api_test_util.cc:416] Found CPU device /job:localhost/replica:0/task:0/device:CPU:0 [ OK ] CAPI.ExecuteAddForward (15 ms) [ RUN ] CAPI.ExecuteAddForwardAsync 2023-04-21 03:35:06.552604: I tensorflow/c/eager/c_api_test_util.cc:416] Found CPU device /job:localhost/replica:0/task:0/device:CPU:0 [ OK ] CAPI.ExecuteAddForwardAsync (2059 ms) [ RUN ] CAPI.Execute_MatMul_CPU [ OK ] CAPI.Execute_MatMul_CPU (42 ms) [ RUN ] CAPI.Execute_MatMul_CPUAsync [ OK ] CAPI.Execute_MatMul_CPUAsync (99 ms) [ RUN ] CAPI.Execute_MatMul_CPU_Runtime_Error [ OK ] CAPI.Execute_MatMul_CPU_Runtime_Error (147 ms) [ RUN ] CAPI.Execute_MatMul_CPU_Runtime_ErrorAsync tensorflow/c/eager/c_api_test.cc:813: Failure Value of: strstr(TF_Message(status), msg) != nullptr Actual: false Expected: true lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [ FAILED ] CAPI.Execute_MatMul_CPU_Runtime_ErrorAsync (100 ms) [ RUN ] CAPI.Execute_MatMul_CPU_Type_Error [ OK ] CAPI.Execute_MatMul_CPU_Type_Error (109 ms) [ RUN ] CAPI.Execute_MatMul_CPU_Type_ErrorAsync [ OK ] CAPI.Execute_MatMul_CPU_Type_ErrorAsync (59 ms) [ RUN ] CAPI.Execute_Min_CPU [ OK ] CAPI.Execute_Min_CPU (21 ms) [ RUN ] CAPI.ExecuteWithTracing [ OK ] CAPI.ExecuteWithTracing (20 ms) [ RUN ] CAPI.ExecuteWithTracingAsync [ OK ] CAPI.ExecuteWithTracingAsync (66 ms) [ RUN ] CAPI.ExecuteOpAndFunctionWithError 2023-04-21 03:35:09.322520: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INTERNAL: Test error. [[{{node error_op}}]] Executing non-communication op originally returned UnavailableError, and was replaced by InternalError to avoid invoking TF network error handling logic. 2023-04-21 03:35:09.322904: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INTERNAL: Test error. [[{{node error_op}}]] Executing non-communication op originally returned UnavailableError, and was replaced by InternalError to avoid invoking TF network error handling logic. 2023-04-21 03:35:09.325883: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): UNAVAILABLE: Test error. [[{{node error_op}}]] 2023-04-21 03:35:09.326150: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): UNAVAILABLE: Test error. [[{{node error_op}}]] [ OK ] CAPI.ExecuteOpAndFunctionWithError (52 ms) [ RUN ] CAPI.FunctionDefAndExecute [ OK ] CAPI.FunctionDefAndExecute (37 ms) [ RUN ] CAPI.FunctionDefAndExecuteAsync [ OK ] CAPI.FunctionDefAndExecuteAsync (23 ms) [ RUN ] CAPI.RunAddFunctionWithGrappler [ OK ] CAPI.RunAddFunctionWithGrappler (31 ms) [ RUN ] CAPI.Variables [ OK ] CAPI.Variables (13 ms) [ RUN ] CAPI.StringAttributes [ OK ] CAPI.StringAttributes (20 ms) [ RUN ] CAPI.TestTFE_SetOpAttrs [ OK ] CAPI.TestTFE_SetOpAttrs (138 ms) [ RUN ] CAPI.TestTFE_TensorHandleCopySharingUnderlyingTensorHandle [ OK ] CAPI.TestTFE_TensorHandleCopySharingUnderlyingTensorHandle (59 ms) [ RUN ] CAPI.TestTFE_OpInferSingleInputAttrs [ OK ] CAPI.TestTFE_OpInferSingleInputAttrs (17 ms) [ RUN ] CAPI.TestTFE_OpInferSingleTypeInputListAttrs [ OK ] CAPI.TestTFE_OpInferSingleTypeInputListAttrs (111 ms) [ RUN ] CAPI.TestTFE_OpInferMixedTypeInputListAttrs [ OK ] CAPI.TestTFE_OpInferMixedTypeInputListAttrs (25 ms) [ RUN ] CAPI.TestTFE_OpAttrsInferenceDisabledWhenNotCallingOpAddInputList [ OK ] CAPI.TestTFE_OpAttrsInferenceDisabledWhenNotCallingOpAddInputList (104 ms) [ RUN ] CAPI.TestTFE_OpGetInputAndOutputLengths [ OK ] CAPI.TestTFE_OpGetInputAndOutputLengths (32 ms) [ RUN ] CAPI.TestTFE_OpGetInputAndOutputLengthsFailForUnknownArguments [ OK ] CAPI.TestTFE_OpGetInputAndOutputLengthsFailForUnknownArguments (76 ms) [ RUN ] CAPI.TestTFE_OpAddAttrs [ OK ] CAPI.TestTFE_OpAddAttrs (59 ms) [ RUN ] CAPI.TestTFE_OpAttrsSerialize [ OK ] CAPI.TestTFE_OpAttrsSerialize (80 ms) [ RUN ] CAPI.TestTFE_OpRecreation [ OK ] CAPI.TestTFE_OpRecreation (16 ms) [ RUN ] CAPI.ShareVariableAcrossContextsWorks 2023-04-21 03:35:10.211040: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:55948 2023-04-21 03:35:10.278061: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:46594 2023-04-21 03:35:10.597526: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:1 2023-04-21 03:35:10.598468: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:10.606666: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:2 2023-04-21 03:35:10.606765: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:10.614126: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:37853 2023-04-21 03:35:10.866404: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:2 2023-04-21 03:35:10.866513: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:10.873186: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:1 2023-04-21 03:35:10.873263: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:10.951398: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:35678 [ OK ] CAPI.ShareVariableAcrossContextsWorks (871 ms) [ RUN ] CAPI.ShareVariableAcrossContextsAfterUpdateContextWorks 2023-04-21 03:35:11.069705: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:46400 2023-04-21 03:35:11.097439: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:60821 2023-04-21 03:35:11.218626: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:2 2023-04-21 03:35:11.218743: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:11.225426: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:1 2023-04-21 03:35:11.225521: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:11.280210: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:43127 2023-04-21 03:35:11.477847: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:2 2023-04-21 03:35:11.477961: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:11.483315: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:1 2023-04-21 03:35:11.483385: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:11.517511: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:52603 2023-04-21 03:35:11.609226: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:41850 2023-04-21 03:35:11.638111: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:1 2023-04-21 03:35:11.638213: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:11.698018: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:1 2023-04-21 03:35:11.698108: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:11.709827: E tensorflow/core/distributed_runtime/rpc/eager/grpc_eager_client.cc:186] Remote EagerContext with id 16151041643620102595 does not seem to exist. [ OK ] CAPI.ShareVariableAcrossContextsAfterUpdateContextWorks (750 ms) [ RUN ] CAPI.SingleHostServerDefWorks 2023-04-21 03:35:11.817675: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:36419 2023-04-21 03:35:11.845360: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:worker/replica:0/task:1 2023-04-21 03:35:11.845444: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:11.854357: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:41668 2023-04-21 03:35:11.924108: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:43378 2023-04-21 03:35:11.983734: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:worker/replica:0/task:1 2023-04-21 03:35:11.983825: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:11.996580: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:worker/replica:0/task:0 2023-04-21 03:35:11.996694: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:12.107084: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:34421 [ OK ] CAPI.SingleHostServerDefWorks (343 ms) [----------] 46 tests from CAPI (6019 ms total) [----------] Global test environment tear-down [==========] 48 tests from 2 test suites ran. (6072 ms total) [ PASSED ] 47 tests. [ FAILED ] 1 test, listed below: [ FAILED ] CAPI.Execute_MatMul_CPU_Runtime_ErrorAsync 1 FAILED TEST YOU HAVE 1 DISABLED TEST ================================================================================ ==================== Test output for //tensorflow/c/eager:c_api_test_cpu: 2023-04-21 03:35:39.718053: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2023-04-21 03:35:39.718230: E tensorflow/tsl/lib/monitoring/collection_registry.cc:81] Cannot register 2 metrics with the same name: /tensorflow/core/bfc_allocator_delay [==========] Running 48 tests from 2 test suites. [----------] Global test environment set-up. [----------] 2 tests from CApiDebug [ RUN ] CApiDebug.ScalarCPU [ OK ] CApiDebug.ScalarCPU (34 ms) [ RUN ] CApiDebug.2DCPU [ OK ] CApiDebug.2DCPU (9 ms) [----------] 2 tests from CApiDebug (44 ms total) [----------] 46 tests from CAPI [ RUN ] CAPI.Context [ OK ] CAPI.Context (13 ms) [ RUN ] CAPI.TensorHandle [ OK ] CAPI.TensorHandle (10 ms) [ RUN ] CAPI.TensorHandleCopyBetweenDevices [ OK ] CAPI.TensorHandleCopyBetweenDevices (8 ms) [ RUN ] CAPI.TensorHandleCopyBetweenDevicesAsync [ OK ] CAPI.TensorHandleCopyBetweenDevicesAsync (9 ms) [ RUN ] CAPI.TensorHandleCopyBetweenDevicesError [ OK ] CAPI.TensorHandleCopyBetweenDevicesError (9 ms) [ RUN ] CAPI.TensorHandleCopyBetweenDevicesErrorAsync [ OK ] CAPI.TensorHandleCopyBetweenDevicesErrorAsync (9 ms) [ RUN ] CAPI.TensorHandleCopyBetweenTwoGPUDevices [ OK ] CAPI.TensorHandleCopyBetweenTwoGPUDevices (22 ms) [ RUN ] CAPI.TensorHandleCopyBetweenTwoGPUDevicesAsync [ OK ] CAPI.TensorHandleCopyBetweenTwoGPUDevicesAsync (9 ms) [ RUN ] CAPI.TensorHandleSilentCopy [ OK ] CAPI.TensorHandleSilentCopy (9 ms) [ RUN ] CAPI.TensorHandleSilentCopyAsync [ OK ] CAPI.TensorHandleSilentCopyAsync (54 ms) [ RUN ] CAPI.TensorHandleSilentCopyLocalPolicy [ OK ] CAPI.TensorHandleSilentCopyLocalPolicy (18 ms) [ RUN ] CAPI.TensorHandleSilentCopyLocalPolicyAsync [ OK ] CAPI.TensorHandleSilentCopyLocalPolicyAsync (10 ms) [ RUN ] CAPI.TensorHandleNullptr [ OK ] CAPI.TensorHandleNullptr (0 ms) [ RUN ] CAPI.TensorHandleDevices [ OK ] CAPI.TensorHandleDevices (52 ms) [ RUN ] CAPI.ExecuteAdd 2023-04-21 03:35:40.075924: I tensorflow/c/eager/c_api_test_util.cc:416] Found CPU device /job:localhost/replica:0/task:0/device:CPU:0 [ OK ] CAPI.ExecuteAdd (33 ms) [ DISABLED ] CAPI.DISABLED_ExecuteAddAsync [ RUN ] CAPI.ExecuteAddForward 2023-04-21 03:35:40.084433: I tensorflow/c/eager/c_api_test_util.cc:416] Found CPU device /job:localhost/replica:0/task:0/device:CPU:0 [ OK ] CAPI.ExecuteAddForward (8 ms) [ RUN ] CAPI.ExecuteAddForwardAsync 2023-04-21 03:35:40.092750: I tensorflow/c/eager/c_api_test_util.cc:416] Found CPU device /job:localhost/replica:0/task:0/device:CPU:0 [ OK ] CAPI.ExecuteAddForwardAsync (1989 ms) [ RUN ] CAPI.Execute_MatMul_CPU [ OK ] CAPI.Execute_MatMul_CPU (23 ms) [ RUN ] CAPI.Execute_MatMul_CPUAsync [ OK ] CAPI.Execute_MatMul_CPUAsync (20 ms) [ RUN ] CAPI.Execute_MatMul_CPU_Runtime_Error [ OK ] CAPI.Execute_MatMul_CPU_Runtime_Error (10 ms) [ RUN ] CAPI.Execute_MatMul_CPU_Runtime_ErrorAsync tensorflow/c/eager/c_api_test.cc:813: Failure Value of: strstr(TF_Message(status), msg) != nullptr Actual: false Expected: true lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [ FAILED ] CAPI.Execute_MatMul_CPU_Runtime_ErrorAsync (27 ms) [ RUN ] CAPI.Execute_MatMul_CPU_Type_Error [ OK ] CAPI.Execute_MatMul_CPU_Type_Error (13 ms) [ RUN ] CAPI.Execute_MatMul_CPU_Type_ErrorAsync [ OK ] CAPI.Execute_MatMul_CPU_Type_ErrorAsync (21 ms) [ RUN ] CAPI.Execute_Min_CPU [ OK ] CAPI.Execute_Min_CPU (12 ms) [ RUN ] CAPI.ExecuteWithTracing [ OK ] CAPI.ExecuteWithTracing (16 ms) [ RUN ] CAPI.ExecuteWithTracingAsync [ OK ] CAPI.ExecuteWithTracingAsync (17 ms) [ RUN ] CAPI.ExecuteOpAndFunctionWithError 2023-04-21 03:35:42.281303: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INTERNAL: Test error. [[{{node error_op}}]] Executing non-communication op originally returned UnavailableError, and was replaced by InternalError to avoid invoking TF network error handling logic. 2023-04-21 03:35:42.281595: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INTERNAL: Test error. [[{{node error_op}}]] Executing non-communication op originally returned UnavailableError, and was replaced by InternalError to avoid invoking TF network error handling logic. 2023-04-21 03:35:42.283723: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): UNAVAILABLE: Test error. [[{{node error_op}}]] 2023-04-21 03:35:42.283918: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): UNAVAILABLE: Test error. [[{{node error_op}}]] [ OK ] CAPI.ExecuteOpAndFunctionWithError (43 ms) [ RUN ] CAPI.FunctionDefAndExecute [ OK ] CAPI.FunctionDefAndExecute (48 ms) [ RUN ] CAPI.FunctionDefAndExecuteAsync [ OK ] CAPI.FunctionDefAndExecuteAsync (29 ms) [ RUN ] CAPI.RunAddFunctionWithGrappler [ OK ] CAPI.RunAddFunctionWithGrappler (23 ms) [ RUN ] CAPI.Variables [ OK ] CAPI.Variables (7 ms) [ RUN ] CAPI.StringAttributes [ OK ] CAPI.StringAttributes (20 ms) [ RUN ] CAPI.TestTFE_SetOpAttrs [ OK ] CAPI.TestTFE_SetOpAttrs (9 ms) [ RUN ] CAPI.TestTFE_TensorHandleCopySharingUnderlyingTensorHandle [ OK ] CAPI.TestTFE_TensorHandleCopySharingUnderlyingTensorHandle (46 ms) [ RUN ] CAPI.TestTFE_OpInferSingleInputAttrs [ OK ] CAPI.TestTFE_OpInferSingleInputAttrs (7 ms) [ RUN ] CAPI.TestTFE_OpInferSingleTypeInputListAttrs [ OK ] CAPI.TestTFE_OpInferSingleTypeInputListAttrs (28 ms) [ RUN ] CAPI.TestTFE_OpInferMixedTypeInputListAttrs [ OK ] CAPI.TestTFE_OpInferMixedTypeInputListAttrs (17 ms) [ RUN ] CAPI.TestTFE_OpAttrsInferenceDisabledWhenNotCallingOpAddInputList [ OK ] CAPI.TestTFE_OpAttrsInferenceDisabledWhenNotCallingOpAddInputList (20 ms) [ RUN ] CAPI.TestTFE_OpGetInputAndOutputLengths [ OK ] CAPI.TestTFE_OpGetInputAndOutputLengths (6 ms) [ RUN ] CAPI.TestTFE_OpGetInputAndOutputLengthsFailForUnknownArguments [ OK ] CAPI.TestTFE_OpGetInputAndOutputLengthsFailForUnknownArguments (28 ms) [ RUN ] CAPI.TestTFE_OpAddAttrs [ OK ] CAPI.TestTFE_OpAddAttrs (13 ms) [ RUN ] CAPI.TestTFE_OpAttrsSerialize [ OK ] CAPI.TestTFE_OpAttrsSerialize (49 ms) [ RUN ] CAPI.TestTFE_OpRecreation [ OK ] CAPI.TestTFE_OpRecreation (80 ms) [ RUN ] CAPI.ShareVariableAcrossContextsWorks 2023-04-21 03:35:42.749810: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:38188 2023-04-21 03:35:42.791340: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:34543 2023-04-21 03:35:42.861621: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:1 2023-04-21 03:35:42.862482: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:42.866411: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:2 2023-04-21 03:35:42.866508: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:42.882273: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:35967 2023-04-21 03:35:42.956553: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:1 2023-04-21 03:35:42.956632: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:42.956668: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:2 2023-04-21 03:35:42.956741: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:42.964030: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:38219 [ OK ] CAPI.ShareVariableAcrossContextsWorks (279 ms) [ RUN ] CAPI.ShareVariableAcrossContextsAfterUpdateContextWorks 2023-04-21 03:35:43.016077: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:47058 2023-04-21 03:35:43.025880: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:59983 2023-04-21 03:35:43.216188: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:1 2023-04-21 03:35:43.216268: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:43.216367: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:2 2023-04-21 03:35:43.216405: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:43.288736: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:32877 2023-04-21 03:35:43.363183: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:1 2023-04-21 03:35:43.363276: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:43.376671: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:2 2023-04-21 03:35:43.376762: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:43.382448: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:55653 2023-04-21 03:35:43.396200: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:39023 2023-04-21 03:35:43.550224: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:1 2023-04-21 03:35:43.550318: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:43.718500: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:1 2023-04-21 03:35:43.718595: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:43.736876: E tensorflow/core/distributed_runtime/rpc/eager/grpc_eager_client.cc:186] Remote EagerContext with id 15521392240502368079 does not seem to exist. [ OK ] CAPI.ShareVariableAcrossContextsAfterUpdateContextWorks (792 ms) [ RUN ] CAPI.SingleHostServerDefWorks 2023-04-21 03:35:43.808916: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:55239 2023-04-21 03:35:43.847422: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:worker/replica:0/task:1 2023-04-21 03:35:43.847508: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:43.855670: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:52838 2023-04-21 03:35:43.884635: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:59464 2023-04-21 03:35:43.937343: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:worker/replica:0/task:0 2023-04-21 03:35:43.937397: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:worker/replica:0/task:1 2023-04-21 03:35:43.937432: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:43.937458: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:35:43.944737: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:53662 [ OK ] CAPI.SingleHostServerDefWorks (148 ms) [----------] 46 tests from CAPI (4140 ms total) [----------] Global test environment tear-down [==========] 48 tests from 2 test suites ran. (4185 ms total) [ PASSED ] 47 tests. [ FAILED ] 1 test, listed below: [ FAILED ] CAPI.Execute_MatMul_CPU_Runtime_ErrorAsync 1 FAILED TEST YOU HAVE 1 DISABLED TEST ================================================================================ ==================== Test output for //tensorflow/c/eager:c_api_test_cpu: 2023-04-21 03:36:14.098682: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2023-04-21 03:36:14.098884: E tensorflow/tsl/lib/monitoring/collection_registry.cc:81] Cannot register 2 metrics with the same name: /tensorflow/core/bfc_allocator_delay [==========] Running 48 tests from 2 test suites. [----------] Global test environment set-up. [----------] 2 tests from CApiDebug [ RUN ] CApiDebug.ScalarCPU [ OK ] CApiDebug.ScalarCPU (48 ms) [ RUN ] CApiDebug.2DCPU [ OK ] CApiDebug.2DCPU (11 ms) [----------] 2 tests from CApiDebug (59 ms total) [----------] 46 tests from CAPI [ RUN ] CAPI.Context [ OK ] CAPI.Context (33 ms) [ RUN ] CAPI.TensorHandle [ OK ] CAPI.TensorHandle (28 ms) [ RUN ] CAPI.TensorHandleCopyBetweenDevices [ OK ] CAPI.TensorHandleCopyBetweenDevices (10 ms) [ RUN ] CAPI.TensorHandleCopyBetweenDevicesAsync [ OK ] CAPI.TensorHandleCopyBetweenDevicesAsync (15 ms) [ RUN ] CAPI.TensorHandleCopyBetweenDevicesError [ OK ] CAPI.TensorHandleCopyBetweenDevicesError (9 ms) [ RUN ] CAPI.TensorHandleCopyBetweenDevicesErrorAsync [ OK ] CAPI.TensorHandleCopyBetweenDevicesErrorAsync (26 ms) [ RUN ] CAPI.TensorHandleCopyBetweenTwoGPUDevices [ OK ] CAPI.TensorHandleCopyBetweenTwoGPUDevices (28 ms) [ RUN ] CAPI.TensorHandleCopyBetweenTwoGPUDevicesAsync [ OK ] CAPI.TensorHandleCopyBetweenTwoGPUDevicesAsync (59 ms) [ RUN ] CAPI.TensorHandleSilentCopy [ OK ] CAPI.TensorHandleSilentCopy (15 ms) [ RUN ] CAPI.TensorHandleSilentCopyAsync [ OK ] CAPI.TensorHandleSilentCopyAsync (33 ms) [ RUN ] CAPI.TensorHandleSilentCopyLocalPolicy [ OK ] CAPI.TensorHandleSilentCopyLocalPolicy (13 ms) [ RUN ] CAPI.TensorHandleSilentCopyLocalPolicyAsync [ OK ] CAPI.TensorHandleSilentCopyLocalPolicyAsync (16 ms) [ RUN ] CAPI.TensorHandleNullptr [ OK ] CAPI.TensorHandleNullptr (0 ms) [ RUN ] CAPI.TensorHandleDevices [ OK ] CAPI.TensorHandleDevices (19 ms) [ RUN ] CAPI.ExecuteAdd 2023-04-21 03:36:14.545763: I tensorflow/c/eager/c_api_test_util.cc:416] Found CPU device /job:localhost/replica:0/task:0/device:CPU:0 [ OK ] CAPI.ExecuteAdd (34 ms) [ DISABLED ] CAPI.DISABLED_ExecuteAddAsync [ RUN ] CAPI.ExecuteAddForward 2023-04-21 03:36:14.554954: I tensorflow/c/eager/c_api_test_util.cc:416] Found CPU device /job:localhost/replica:0/task:0/device:CPU:0 [ OK ] CAPI.ExecuteAddForward (15 ms) [ RUN ] CAPI.ExecuteAddForwardAsync 2023-04-21 03:36:14.570695: I tensorflow/c/eager/c_api_test_util.cc:416] Found CPU device /job:localhost/replica:0/task:0/device:CPU:0 [ OK ] CAPI.ExecuteAddForwardAsync (2090 ms) [ RUN ] CAPI.Execute_MatMul_CPU [ OK ] CAPI.Execute_MatMul_CPU (79 ms) [ RUN ] CAPI.Execute_MatMul_CPUAsync [ OK ] CAPI.Execute_MatMul_CPUAsync (28 ms) [ RUN ] CAPI.Execute_MatMul_CPU_Runtime_Error [ OK ] CAPI.Execute_MatMul_CPU_Runtime_Error (14 ms) [ RUN ] CAPI.Execute_MatMul_CPU_Runtime_ErrorAsync tensorflow/c/eager/c_api_test.cc:813: Failure Value of: strstr(TF_Message(status), msg) != nullptr Actual: false Expected: true lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [ FAILED ] CAPI.Execute_MatMul_CPU_Runtime_ErrorAsync (11 ms) [ RUN ] CAPI.Execute_MatMul_CPU_Type_Error [ OK ] CAPI.Execute_MatMul_CPU_Type_Error (19 ms) [ RUN ] CAPI.Execute_MatMul_CPU_Type_ErrorAsync [ OK ] CAPI.Execute_MatMul_CPU_Type_ErrorAsync (45 ms) [ RUN ] CAPI.Execute_Min_CPU [ OK ] CAPI.Execute_Min_CPU (52 ms) [ RUN ] CAPI.ExecuteWithTracing [ OK ] CAPI.ExecuteWithTracing (10 ms) [ RUN ] CAPI.ExecuteWithTracingAsync [ OK ] CAPI.ExecuteWithTracingAsync (9 ms) [ RUN ] CAPI.ExecuteOpAndFunctionWithError 2023-04-21 03:36:16.960547: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INTERNAL: Test error. [[{{node error_op}}]] Executing non-communication op originally returned UnavailableError, and was replaced by InternalError to avoid invoking TF network error handling logic. 2023-04-21 03:36:16.960815: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INTERNAL: Test error. [[{{node error_op}}]] Executing non-communication op originally returned UnavailableError, and was replaced by InternalError to avoid invoking TF network error handling logic. 2023-04-21 03:36:16.962891: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): UNAVAILABLE: Test error. [[{{node error_op}}]] 2023-04-21 03:36:16.963074: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): UNAVAILABLE: Test error. [[{{node error_op}}]] [ OK ] CAPI.ExecuteOpAndFunctionWithError (39 ms) [ RUN ] CAPI.FunctionDefAndExecute [ OK ] CAPI.FunctionDefAndExecute (33 ms) [ RUN ] CAPI.FunctionDefAndExecuteAsync [ OK ] CAPI.FunctionDefAndExecuteAsync (16 ms) [ RUN ] CAPI.RunAddFunctionWithGrappler [ OK ] CAPI.RunAddFunctionWithGrappler (19 ms) [ RUN ] CAPI.Variables [ OK ] CAPI.Variables (6 ms) [ RUN ] CAPI.StringAttributes [ OK ] CAPI.StringAttributes (11 ms) [ RUN ] CAPI.TestTFE_SetOpAttrs [ OK ] CAPI.TestTFE_SetOpAttrs (8 ms) [ RUN ] CAPI.TestTFE_TensorHandleCopySharingUnderlyingTensorHandle [ OK ] CAPI.TestTFE_TensorHandleCopySharingUnderlyingTensorHandle (25 ms) [ RUN ] CAPI.TestTFE_OpInferSingleInputAttrs [ OK ] CAPI.TestTFE_OpInferSingleInputAttrs (105 ms) [ RUN ] CAPI.TestTFE_OpInferSingleTypeInputListAttrs [ OK ] CAPI.TestTFE_OpInferSingleTypeInputListAttrs (109 ms) [ RUN ] CAPI.TestTFE_OpInferMixedTypeInputListAttrs [ OK ] CAPI.TestTFE_OpInferMixedTypeInputListAttrs (40 ms) [ RUN ] CAPI.TestTFE_OpAttrsInferenceDisabledWhenNotCallingOpAddInputList [ OK ] CAPI.TestTFE_OpAttrsInferenceDisabledWhenNotCallingOpAddInputList (89 ms) [ RUN ] CAPI.TestTFE_OpGetInputAndOutputLengths [ OK ] CAPI.TestTFE_OpGetInputAndOutputLengths (15 ms) [ RUN ] CAPI.TestTFE_OpGetInputAndOutputLengthsFailForUnknownArguments [ OK ] CAPI.TestTFE_OpGetInputAndOutputLengthsFailForUnknownArguments (12 ms) [ RUN ] CAPI.TestTFE_OpAddAttrs [ OK ] CAPI.TestTFE_OpAddAttrs (16 ms) [ RUN ] CAPI.TestTFE_OpAttrsSerialize [ OK ] CAPI.TestTFE_OpAttrsSerialize (9 ms) [ RUN ] CAPI.TestTFE_OpRecreation [ OK ] CAPI.TestTFE_OpRecreation (11 ms) [ RUN ] CAPI.ShareVariableAcrossContextsWorks 2023-04-21 03:36:17.519854: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:48235 2023-04-21 03:36:17.584392: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:51407 2023-04-21 03:36:17.779829: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:2 2023-04-21 03:36:17.779890: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:1 2023-04-21 03:36:17.780689: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:36:17.780714: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:36:17.786587: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:40807 2023-04-21 03:36:17.817332: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:2 2023-04-21 03:36:17.817406: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:36:17.826544: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:1 2023-04-21 03:36:17.826610: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:36:17.856745: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:33837 [ OK ] CAPI.ShareVariableAcrossContextsWorks (368 ms) [ RUN ] CAPI.ShareVariableAcrossContextsAfterUpdateContextWorks 2023-04-21 03:36:17.980496: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:50289 2023-04-21 03:36:17.990663: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:39689 2023-04-21 03:36:18.066585: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:2 2023-04-21 03:36:18.087023: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:36:18.092024: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:1 2023-04-21 03:36:18.092105: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:36:18.116494: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:52872 2023-04-21 03:36:18.136952: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:2 2023-04-21 03:36:18.137033: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:36:18.141903: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:1 2023-04-21 03:36:18.141953: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:36:18.148116: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:43960 2023-04-21 03:36:18.256422: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:59005 2023-04-21 03:36:18.326770: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:1 2023-04-21 03:36:18.326850: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:36:18.357509: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:localhost/replica:0/task:1 2023-04-21 03:36:18.357591: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:36:18.363836: E tensorflow/core/distributed_runtime/rpc/eager/grpc_eager_client.cc:186] Remote EagerContext with id 13677449324157915715 does not seem to exist. [ OK ] CAPI.ShareVariableAcrossContextsAfterUpdateContextWorks (536 ms) [ RUN ] CAPI.SingleHostServerDefWorks 2023-04-21 03:36:18.417764: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:46434 2023-04-21 03:36:18.456663: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:worker/replica:0/task:1 2023-04-21 03:36:18.456748: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:36:18.462199: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:40337 2023-04-21 03:36:18.539527: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:39201 2023-04-21 03:36:18.647166: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:worker/replica:0/task:1 2023-04-21 03:36:18.647255: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:36:18.647512: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:302] Creating sync eager service context with rendezvous_id on host ldcg-aarch64-02 /job:worker/replica:0/task:0 2023-04-21 03:36:18.647547: I tensorflow/core/distributed_runtime/eager/eager_service_impl.cc:308] SessionOptions: 2023-04-21 03:36:18.654602: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:57103 [ OK ] CAPI.SingleHostServerDefWorks (250 ms) [----------] 46 tests from CAPI (4454 ms total) [----------] Global test environment tear-down [==========] 48 tests from 2 test suites ran. (4515 ms total) [ PASSED ] 47 tests. [ FAILED ] 1 test, listed below: [ FAILED ] CAPI.Execute_MatMul_CPU_Runtime_ErrorAsync 1 FAILED TEST YOU HAVE 1 DISABLED TEST ================================================================================ ==================== Test output for //tensorflow/dtensor/python/tests:input_util_test (shard 1 of 8): 2023-04-21 03:40:36.687131: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. Running tests under Python 3.8.10: /usr/local/bin/python3 [ RUN ] DTensorDatasetTest.testForInIterationEager I0421 03:40:41.036333 281473008956256 mesh_util.py:35] This is client 0 of 1 clients I0421 03:40:41.036572 281473008956256 mesh_util.py:36] Number of global CPU devices: 16 I0421 03:40:41.036732 281473008956256 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0421 03:40:41.037261 281473008956256 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0421 03:40:41.037441 281473008956256 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-04-21 03:40:41.366393: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_0}}]] 2023-04-21 03:40:41.417886: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-04-21 03:40:41.474917: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-04-21 03:40:41.527617: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-04-21 03:40:41.579462: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-04-21 03:40:41.619124: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-04-21 03:40:41.619852: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:41.697710: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-04-21 03:40:41.698492: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:41.902328: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:41.903060: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:41.961405: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:41.962217: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-04-21 03:40:42.037204: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-04-21 03:40:42.037922: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:42.108310: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:42.109107: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:42.209513: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:42.210241: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:42.309995: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-04-21 03:40:42.310807: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-04-21 03:40:42.406428: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:42.407178: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-04-21 03:40:42.538728: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-04-21 03:40:42.539447: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-04-21 03:40:42.614666: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:42.615444: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-04-21 03:40:42.671713: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:42.672527: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-04-21 03:40:42.733295: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-04-21 03:40:42.734106: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-04-21 03:40:42.819068: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-04-21 03:40:42.819892: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-04-21 03:40:42.894104: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-04-21 03:40:42.894931: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:42.965603: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [32,8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:42.966466: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_14' with dtype int32 and shape [3,4] [[{{node Placeholder/_14}}]] 2023-04-21 03:40:43.622108: E tensorflow/dtensor/cc/dtensor_device.cc:2168] Encountered error while executing function: CopyToMesh__func_11958644085055432363_5950845106898307211_6858324592425701659_1 for mesh : |batch=4,height=2,width=2|0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15|0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15|/job:localhost/replica:0/task:0/device:CPU:0,/job:localhost/replica:0/task:0/device:CPU:1,/job:localhost/replica:0/task:0/device:CPU:2,/job:localhost/replica:0/task:0/device:CPU:3,/job:localhost/replica:0/task:0/device:CPU:4,/job:localhost/replica:0/task:0/device:CPU:5,/job:localhost/replica:0/task:0/device:CPU:6,/job:localhost/replica:0/task:0/device:CPU:7,/job:localhost/replica:0/task:0/device:CPU:8,/job:localhost/replica:0/task:0/device:CPU:9,/job:localhost/replica:0/task:0/device:CPU:10,/job:localhost/replica:0/task:0/device:CPU:11,/job:localhost/replica:0/task:0/device:CPU:12,/job:localhost/replica:0/task:0/device:CPU:13,/job:localhost/replica:0/task:0/device:CPU:14,/job:localhost/replica:0/task:0/device:CPU:15 / error : {{function_node IteratorGetNext__func_8392981028614849161_12989052755378955192_1628789561242421788_0}} End of sequence [[{{node tf.StatefulPartitionedCall/eager_operation}}]] Encountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors. 2023-04-21 03:40:43.622338: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:9] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Operation was cancelled [[{{node tf.StatefulPartitionedCall/eager_operation}}]] 2023-04-21 03:40:43.622656: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:2] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Operation was cancelled [[{{node tf.StatefulPartitionedCall/eager_operation}}]] 2023-04-21 03:40:43.626200: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:14] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): CANCELLED: Operation was cancelled [[{{node tf.StatefulPartitionedCall/eager_operation}}]] 2023-04-21 03:40:43.636519: E tensorflow/dtensor/cc/dtensor_device.cc:2202] Error executing CopyToMesh {{function_node IteratorGetNext__func_8392981028614849161_12989052755378955192_1628789561242421788_0}} End of sequence [[{{node tf.StatefulPartitionedCall/eager_operation}}]] Encountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors. INFO:tensorflow:time(__main__.DTensorDatasetTest.testForInIterationEager): 2.75s I0421 03:40:43.789635 281473008956256 test_util.py:2462] time(__main__.DTensorDatasetTest.testForInIterationEager): 2.75s [ FAILED ] DTensorDatasetTest.testForInIterationEager [ RUN ] DTensorDatasetTest.testIterOnBatchedDatasetGraph I0421 03:40:43.793795 281473008956256 mesh_util.py:35] This is client 0 of 1 clients I0421 03:40:43.794048 281473008956256 mesh_util.py:36] Number of global CPU devices: 16 I0421 03:40:43.794218 281473008956256 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0421 03:40:43.794748 281473008956256 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0421 03:40:43.794899 281473008956256 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-04-21 03:40:44.131866: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-04-21 03:40:44.210243: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-04-21 03:40:44.284758: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-04-21 03:40:44.361612: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-04-21 03:40:44.422813: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:44.423797: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:44.509482: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:44.510499: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-04-21 03:40:44.617780: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:44.618802: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-04-21 03:40:44.696341: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:44.697366: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:44.846350: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-04-21 03:40:44.847306: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-04-21 03:40:45.030052: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-04-21 03:40:45.031021: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:45.151157: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:45.152207: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:45.262162: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:45.263238: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:45.350696: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:45.351702: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-04-21 03:40:45.428308: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:45.429293: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:45.528696: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:45.529747: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:45.610990: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:45.612655: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:45.739224: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:45.740232: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:45.829178: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:45.830127: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:45.941308: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:45.942354: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:46.141077: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:46.142117: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:46.382852: I tensorflow/dtensor/cc/dtensor_device.cc:1559] DTensor cache key lookup missed for __inference_train_636. DTensor is (re-)computing its SPMD transformation. 2023-04-21 03:40:46.391860: I tensorflow/dtensor/cc/dtensor_device.cc:1628] DTensor cache key lookup missed for __inference_train_636. DTensor is (re-)computing its ExecutionFunctions. 2023-04-21 03:40:46.589009: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_1' with dtype float and shape [1] [[{{node Placeholder/_1}}]] INFO:tensorflow:time(__main__.DTensorDatasetTest.testIterOnBatchedDatasetGraph): 3.66s I0421 03:40:47.454980 281473008956256 test_util.py:2462] time(__main__.DTensorDatasetTest.testIterOnBatchedDatasetGraph): 3.66s [ OK ] DTensorDatasetTest.testIterOnBatchedDatasetGraph [ RUN ] DTensorDatasetTest.testIterWithLayouts1 (images_sharding=['unsharded', 'unsharded', 'unsharded', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=True) I0421 03:40:47.457370 281473008956256 mesh_util.py:35] This is client 0 of 1 clients I0421 03:40:47.457612 281473008956256 mesh_util.py:36] Number of global CPU devices: 16 I0421 03:40:47.457789 281473008956256 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0421 03:40:47.458302 281473008956256 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0421 03:40:47.458458 281473008956256 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-04-21 03:40:47.787082: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-04-21 03:40:47.854118: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:47.855203: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:47.972348: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:47.973401: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:48.234673: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-04-21 03:40:48.235704: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:48.311482: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:48.312491: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:48.402150: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:48.403243: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-04-21 03:40:48.501943: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:48.503006: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-04-21 03:40:48.593011: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:48.594077: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-04-21 03:40:48.690106: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-04-21 03:40:48.691154: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-04-21 03:40:48.803796: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:48.804830: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-04-21 03:40:48.884259: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:48.885330: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:48.974192: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:48.975264: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-04-21 03:40:49.115486: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:49.119122: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-04-21 03:40:49.193358: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:49.194352: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:49.270777: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:49.271816: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:49.367989: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:49.368952: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:49.469700: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:49.470716: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-04-21 03:40:49.613929: I tensorflow/dtensor/cc/dtensor_device.cc:1559] DTensor cache key lookup missed for __inference_train_873. DTensor is (re-)computing its SPMD transformation. 2023-04-21 03:40:49.622578: I tensorflow/dtensor/cc/dtensor_device.cc:1628] DTensor cache key lookup missed for __inference_train_873. DTensor is (re-)computing its ExecutionFunctions. 2023-04-21 03:40:49.860485: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_1' with dtype float and shape [1] [[{{node Placeholder/_1}}]] INFO:tensorflow:time(__main__.DTensorDatasetTest.testIterWithLayouts1 (images_sharding=['unsharded', 'unsharded', 'unsharded', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=True)): 3.38s I0421 03:40:50.839580 281473008956256 test_util.py:2462] time(__main__.DTensorDatasetTest.testIterWithLayouts1 (images_sharding=['unsharded', 'unsharded', 'unsharded', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=True)): 3.38s [ OK ] DTensorDatasetTest.testIterWithLayouts1 (images_sharding=['unsharded', 'unsharded', 'unsharded', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=True) [ RUN ] DTensorDatasetTest.testIterWithLayouts7 (images_sharding=['unsharded', 'width', 'height', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=True) I0421 03:40:50.841269 281473008956256 mesh_util.py:35] This is client 0 of 1 clients I0421 03:40:50.841468 281473008956256 mesh_util.py:36] Number of global CPU devices: 16 I0421 03:40:50.841640 281473008956256 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3]] [[ 4 5] [ 6 7]] [[ 8 9] [10 11]] [[12 13] [14 15]]] I0421 03:40:50.842145 281473008956256 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0421 03:40:50.842309 281473008956256 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] 2023-04-21 03:40:51.148710: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-04-21 03:40:51.217053: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:51.218053: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:51.338754: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:51.339783: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:51.582941: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:51.584024: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-04-21 03:40:51.686473: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:51.687553: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:51.813129: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:51.814276: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:52.005395: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:52.006568: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:52.098205: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:52.099349: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_18' with dtype int32 and shape [3,2] [[{{node Placeholder/_18}}]] 2023-04-21 03:40:52.241105: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:52.242171: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:52.389866: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:52.390947: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:52.482489: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:52.483584: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-04-21 03:40:52.614905: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:52.615952: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:52.712177: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:52.713180: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:52.828909: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:52.829980: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-04-21 03:40:52.923610: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:52.924700: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:53.064919: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_5' with dtype float and shape [1] [[{{node Placeholder/_5}}]] 2023-04-21 03:40:53.065945: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_17' with dtype int32 and shape [3,4] [[{{node Placeholder/_17}}]] 2023-04-21 03:40:53.159056: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:53.160096: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:53.261560: I tensorflow/dtensor/cc/dtensor_device.cc:1559] DTensor cache key lookup missed for __inference_train_1114. DTensor is (re-)computing its SPMD transformation. 2023-04-21 03:40:53.269878: I tensorflow/dtensor/cc/dtensor_device.cc:1628] DTensor cache key lookup missed for __inference_train_1114. DTensor is (re-)computing its ExecutionFunctions. 2023-04-21 03:40:53.455913: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_1' with dtype float and shape [1] [[{{node Placeholder/_1}}]] INFO:tensorflow:time(__main__.DTensorDatasetTest.testIterWithLayouts7 (images_sharding=['unsharded', 'width', 'height', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=True)): 3.33s I0421 03:40:54.173009 281473008956256 test_util.py:2462] time(__main__.DTensorDatasetTest.testIterWithLayouts7 (images_sharding=['unsharded', 'width', 'height', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=True)): 3.33s [ OK ] DTensorDatasetTest.testIterWithLayouts7 (images_sharding=['unsharded', 'width', 'height', 'unsharded'], labels_sharding=['unsharded', 'unsharded'], is_graph=True) [ RUN ] DTensorIteratorSpecTest.testFromTensorList I0421 03:40:54.175011 281473008956256 mesh_util.py:35] This is client 0 of 1 clients I0421 03:40:54.175222 281473008956256 mesh_util.py:36] Number of global CPU devices: 8 I0421 03:40:54.175405 281473008956256 mesh_util.py:39] Global device IDs: [0 1 2 3 4 5 6 7] I0421 03:40:54.175827 281473008956256 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7] I0421 03:40:54.175995 281473008956256 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7'] 2023-04-21 03:40:54.408874: I tensorflow/core/grappler/optimizers/data/replicate_on_split.cc:32] Running replicate on split optimization 2023-04-21 03:40:54.464000: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:54.464961: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:54.562526: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_16' with dtype int32 and shape [1,4] [[{{node Placeholder/_16}}]] 2023-04-21 03:40:54.563444: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_16' with dtype int32 and shape [1,4] [[{{node Placeholder/_16}}]] 2023-04-21 03:40:54.649003: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:54.649953: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:54.722778: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_16' with dtype int32 and shape [1,4] [[{{node Placeholder/_16}}]] 2023-04-21 03:40:54.723721: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_16' with dtype int32 and shape [1,4] [[{{node Placeholder/_16}}]] 2023-04-21 03:40:54.798153: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:54.799060: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:54.876282: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:54.877223: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_16' with dtype int32 and shape [1,4] [[{{node Placeholder/_16}}]] 2023-04-21 03:40:54.950244: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] 2023-04-21 03:40:54.951190: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_16' with dtype int32 and shape [1,4] [[{{node Placeholder/_16}}]] 2023-04-21 03:40:55.031780: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_16' with dtype int32 and shape [1,4] [[{{node Placeholder/_16}}]] 2023-04-21 03:40:55.032700: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_4' with dtype float and shape [8,8,3] [[{{node Placeholder/_4}}]] INFO:tensorflow:time(__main__.DTensorIteratorSpecTest.testFromTensorList): 1.35s I0421 03:40:55.519832 281473008956256 test_util.py:2462] time(__main__.DTensorIteratorSpecTest.testFromTensorList): 1.35s [ OK ] DTensorIteratorSpecTest.testFromTensorList [ RUN ] InputUtilHelpersTest.testShardCounts5 (mesh_dims=[('batch', 2), ('height', 4), ('width', 2)], layout_specs=['batch', 'width', 'height'], batch_dim='batch', counts=[1, 2, 4]) I0421 03:40:55.522614 281473008956256 mesh_util.py:35] This is client 0 of 1 clients I0421 03:40:55.522864 281473008956256 mesh_util.py:36] Number of global CPU devices: 16 I0421 03:40:55.523053 281473008956256 mesh_util.py:39] Global device IDs: [[[ 0 1] [ 2 3] [ 4 5] [ 6 7]] [[ 8 9] [10 11] [12 13] [14 15]]] I0421 03:40:55.523569 281473008956256 mesh_util.py:40] Local device IDs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] I0421 03:40:55.523741 281473008956256 mesh_util.py:41] Local devices: ['/job:localhost/replica:0/task:0/device:CPU:0', '/job:localhost/replica:0/task:0/device:CPU:1', '/job:localhost/replica:0/task:0/device:CPU:2', '/job:localhost/replica:0/task:0/device:CPU:3', '/job:localhost/replica:0/task:0/device:CPU:4', '/job:localhost/replica:0/task:0/device:CPU:5', '/job:localhost/replica:0/task:0/device:CPU:6', '/job:localhost/replica:0/task:0/device:CPU:7', '/job:localhost/replica:0/task:0/device:CPU:8', '/job:localhost/replica:0/task:0/device:CPU:9', '/job:localhost/replica:0/task:0/device:CPU:10', '/job:localhost/replica:0/task:0/device:CPU:11', '/job:localhost/replica:0/task:0/device:CPU:12', '/job:localhost/replica:0/task:0/device:CPU:13', '/job:localhost/replica:0/task:0/device:CPU:14', '/job:localhost/replica:0/task:0/device:CPU:15'] INFO:tensorflow:time(__main__.InputUtilHelpersTest.testShardCounts5 (mesh_dims=[('batch', 2), ('height', 4), ('width', 2)], layout_specs=['batch', 'width', 'height'], batch_dim='batch', counts=[1, 2, 4])): 0.0s I0421 03:40:55.524475 281473008956256 test_util.py:2462] time(__main__.InputUtilHelpersTest.testShardCounts5 (mesh_dims=[('batch', 2), ('height', 4), ('width', 2)], layout_specs=['batch', 'width', 'height'], batch_dim='batch', counts=[1, 2, 4])): 0.0s [ OK ] InputUtilHelpersTest.testShardCounts5 (mesh_dims=[('batch', 2), ('height', 4), ('width', 2)], layout_specs=['batch', 'width', 'height'], batch_dim='batch', counts=[1, 2, 4]) ====================================================================== ERROR: testForInIterationEager (__main__.DTensorDatasetTest) DTensorDatasetTest.testForInIterationEager testForInIterationEager(False) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/absl_py/absl/testing/parameterized.py", line 316, in bound_param_test return test_method(self, *testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/dtensor/python/tests/input_util_test.py", line 186, in testForInIteration self.assertDTensorEqual(output, images_layout, d_output) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/dtensor/python/tests/test_util.py", line 303, in assertDTensorEqual unpacked = [t.numpy() for t in api.unpack(result_dtensor)] File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/dtensor/python/api.py", line 352, in unpack return _dtensor_device().unpack(tensor) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/dtensor/python/tests/input_util_test.runfiles/org_tensorflow/tensorflow/dtensor/python/dtensor_device.py", line 250, in unpack tensors = _pywrap_dtensor_device.Unpack( ValueError: cannot create std::vector larger than max_size() ---------------------------------------------------------------------- Ran 6 tests in 14.490s FAILED (errors=1) ================================================================================ ==================== Test output for //tensorflow/python/distribute/failure_handling:gce_failure_handler_test (shard 3 of 8): 2023-04-21 03:48:20.451384: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. Running tests under Python 3.8.10: /usr/local/bin/python3 [ RUN ] GceFailureHandlingTest.test_basic_run_test_inputarg_checkpoint_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 18979 I0421 03:48:32.076745 281472889680736 test_util.py:3794] Using local port 18979 INFO:tensorflow:Using local port 19449 I0421 03:48:32.077363 281472889680736 test_util.py:3794] Using local port 19449 INFO:tensorflow:Using local port 19921 I0421 03:48:32.077762 281472889680736 test_util.py:3794] Using local port 19921 INFO:tensorflow:Using local port 24611 I0421 03:48:32.078137 281472889680736 test_util.py:3794] Using local port 24611 2023-04-21 03:48:32.989110: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. 2023-04-21 03:48:33.059613: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. INFO:tensorflow:Cluster starting. I0421 03:48:34.952063 281472889680736 gce_failure_handler_test.py:317] Cluster starting. [worker-0]: I0421 03:48:35.063067 281472986018656 multi_process_runner.py:840] Subprocess with PID 1883851 (worker, 0) is now being started. [worker-1]: I0421 03:48:35.065971 281472986018656 multi_process_runner.py:840] Subprocess with PID 1883890 (worker, 1) is now being started. [worker-0]: I0421 03:48:35.063593 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:18979", "localhost:19449", "localhost:19921", "localhost:24611"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-2]: I0421 03:48:35.070792 281472986018656 multi_process_runner.py:840] Subprocess with PID 1884035 (worker, 2) is now being started. [worker-0]: E0421 03:48:35.087478068 1883851 server_chttp2.cc:40] {"created":"@1682048915.087362581","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1682048915.087357146","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1682048915.087329638","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1682048915.087318967","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1682048915.087356391","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1682048915.087348975","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]} [worker-0]: 2023-04-21 03:48:35.087602: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:599] UNKNOWN: Could not start gRPC server [worker-0]: 2023-04-21 03:48:35.087956: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:699] Could not start gRPC server [worker-1]: I0421 03:48:35.066509 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:18979", "localhost:19449", "localhost:19921", "localhost:24611"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I0421 03:48:35.071289 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:18979", "localhost:19449", "localhost:19921", "localhost:24611"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0421 03:48:35.106142 281472986018656 multi_process_runner.py:840] Subprocess with PID 1884161 (worker, 3) is now being started. [worker-3]: I0421 03:48:35.106545 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:18979", "localhost:19449", "localhost:19921", "localhost:24611"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: Process _Process-2: [worker-0]: Traceback (most recent call last): [worker-0]: File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap [worker-0]: self.run() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 755, in _run_with_setenv [worker-0]: return self._actual_run() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl [worker-0]: app.run(lambda _: self._run_impl()) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run [worker-0]: _run_main(main, args) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main [worker-0]: sys.exit(main(argv)) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in [worker-0]: app.run(lambda _: self._run_impl()) [worker-0]: File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 866, in __call__ [worker-0]: six.reraise(*info.exc_info) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise [worker-0]: raise value [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained [worker-0]: return_value = fn(*args, **kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 134, in worker_fn [worker-0]: strategy = collective_all_reduce_strategy.CollectiveAllReduceStrategy() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 187, in __init__ [worker-0]: CollectiveAllReduceExtended( [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 339, in __init__ [worker-0]: self._initialize_strategy(self._cluster_resolver, devices=devices) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 358, in _initialize_strategy [worker-0]: self._initialize_multi_worker(cluster_resolver) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 530, in _initialize_multi_worker [worker-0]: context.context().ensure_initialized() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 608, in ensure_initialized [worker-0]: pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) [worker-0]: tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server [worker-3]: 2023-04-21 03:48:35.162709: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:24611 [worker-1]: 2023-04-21 03:48:35.247800: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:19449 [worker-2]: 2023-04-21 03:48:35.286913: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:19921 INFO:tensorflow:restarting workers I0421 03:49:05.234528 281472889680736 gce_failure_handler_test.py:323] restarting workers INFO:tensorflow:workers restarted I0421 03:49:05.306732 281472889680736 gce_failure_handler_test.py:327] workers restarted [worker-0]: I0421 03:49:05.339053 281472986018656 multi_process_runner.py:840] Subprocess with PID 1937619 (worker, 0) is now being started. [worker-0]: I0421 03:49:05.339555 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:18979", "localhost:19449", "localhost:19921", "localhost:24611"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-2]: I0421 03:49:05.423482 281472986018656 multi_process_runner.py:840] Subprocess with PID 1938043 (worker, 2) is now being started. [worker-3]: I0421 03:49:05.423419 281472986018656 multi_process_runner.py:840] Subprocess with PID 1938113 (worker, 3) is now being started. [worker-2]: I0421 03:49:05.423893 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:18979", "localhost:19449", "localhost:19921", "localhost:24611"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0421 03:49:05.423895 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:18979", "localhost:19449", "localhost:19921", "localhost:24611"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-1]: I0421 03:49:05.442823 281472986018656 multi_process_runner.py:840] Subprocess with PID 1937656 (worker, 1) is now being started. [worker-1]: I0421 03:49:05.443288 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:18979", "localhost:19449", "localhost:19921", "localhost:24611"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-3]: E0421 03:49:05.458704682 1938113 server_chttp2.cc:40] {"created":"@1682048945.458590545","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1682048945.458584434","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1682048945.458556096","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1682048945.458543945","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1682048945.458583624","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1682048945.458576489","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]} [worker-3]: 2023-04-21 03:49:05.458852: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:599] UNKNOWN: Could not start gRPC server [worker-3]: 2023-04-21 03:49:05.459210: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:699] Could not start gRPC server [worker-1]: E0421 03:49:05.466992966 1937656 server_chttp2.cc:40] {"created":"@1682048945.466879709","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1682048945.466874104","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1682048945.466845996","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1682048945.466834579","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1682048945.466873409","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1682048945.466866278","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]} [worker-1]: 2023-04-21 03:49:05.467122: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:599] UNKNOWN: Could not start gRPC server [worker-1]: 2023-04-21 03:49:05.467715: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:699] Could not start gRPC server [worker-0]: 2023-04-21 03:49:05.489676: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:18979 [worker-2]: E0421 03:49:05.500239297 1938043 server_chttp2.cc:40] {"created":"@1682048945.500127790","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1682048945.500123345","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":341,"referenced_errors":[{"created":"@1682048945.500095322","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1682048945.500083461","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]},{"created":"@1682048945.500122530","description":"Unable to configure socket","fd":9,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":215,"referenced_errors":[{"created":"@1682048945.500115094","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]} [worker-2]: 2023-04-21 03:49:05.500375: E tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:599] UNKNOWN: Could not start gRPC server [worker-2]: 2023-04-21 03:49:05.500752: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:699] Could not start gRPC server [worker-0]: 2023-04-21 03:49:05.546338: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 12908599997081445987 [worker-0]: 2023-04-21 03:49:05.547305: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-3]: Process _Process-9: [worker-3]: Traceback (most recent call last): [worker-3]: File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap [worker-3]: self.run() [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 755, in _run_with_setenv [worker-3]: return self._actual_run() [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl [worker-3]: app.run(lambda _: self._run_impl()) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run [worker-3]: _run_main(main, args) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main [worker-3]: sys.exit(main(argv)) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in [worker-0]: 2023-04-21 03:49:05.776841: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 17827052963074449808 [worker-3]: app.run(lambda _: self._run_impl()) [worker-3]: 2023-04-21 03:49:05.777219: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-3]: File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 866, in __call__ [worker-3]: six.reraise(*info.exc_info) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise [worker-3]: raise value [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained [worker-3]: return_value = fn(*args, **kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 134, in worker_fn [worker-3]: strategy = collective_all_reduce_strategy.CollectiveAllReduceStrategy() [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 187, in __init__ [worker-3]: CollectiveAllReduceExtended( [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 339, in __init__ [worker-3]: self._initialize_strategy(self._cluster_resolver, devices=devices) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 358, in _initialize_strategy [worker-3]: self._initialize_multi_worker(cluster_resolver) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 530, in _initialize_multi_worker [worker-3]: context.context().ensure_initialized() [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 608, in ensure_initialized [worker-3]: pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) [worker-3]: tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server [worker-1]: Process _Process-7: [worker-1]: Traceback (most recent call last): [worker-1]: File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap [worker-1]: self.run() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 755, in _run_with_setenv [worker-1]: return self._actual_run() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl [worker-1]: app.run(lambda _: self._run_impl()) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run [worker-1]: _run_main(main, args) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main [worker-1]: sys.exit(main(argv)) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in [worker-1]: app.run(lambda _: self._run_impl()) [worker-1]: File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 866, in __call__ [worker-1]: six.reraise(*info.exc_info) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise [worker-1]: raise value [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained [worker-1]: return_value = fn(*args, **kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 134, in worker_fn [worker-1]: strategy = collective_all_reduce_strategy.CollectiveAllReduceStrategy() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 187, in __init__ [worker-1]: CollectiveAllReduceExtended( [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 339, in __init__ [worker-1]: self._initialize_strategy(self._cluster_resolver, devices=devices) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 358, in _initialize_strategy [worker-1]: self._initialize_multi_worker(cluster_resolver) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 530, in _initialize_multi_worker [worker-1]: context.context().ensure_initialized() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 608, in ensure_initialized [worker-1]: pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) [worker-1]: tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server [worker-2]: Process _Process-8: [worker-2]: Traceback (most recent call last): [worker-2]: File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap [worker-2]: self.run() [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 755, in _run_with_setenv [worker-2]: return self._actual_run() [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl [worker-2]: app.run(lambda _: self._run_impl()) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run [worker-2]: _run_main(main, args) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main [worker-2]: sys.exit(main(argv)) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in [worker-2]: app.run(lambda _: self._run_impl()) [worker-2]: File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run [worker-2]: self._target(*self._args, **self._kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 866, in __call__ [worker-2]: six.reraise(*info.exc_info) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise [worker-2]: raise value [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained [worker-2]: return_value = fn(*args, **kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 134, in worker_fn [worker-2]: strategy = collective_all_reduce_strategy.CollectiveAllReduceStrategy() [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 187, in __init__ [worker-2]: CollectiveAllReduceExtended( [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 339, in __init__ [worker-2]: self._initialize_strategy(self._cluster_resolver, devices=devices) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 358, in _initialize_strategy [worker-2]: self._initialize_multi_worker(cluster_resolver) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 530, in _initialize_multi_worker [worker-2]: context.context().ensure_initialized() [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 608, in ensure_initialized [worker-2]: pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str) [worker-2]: tensorflow.python.framework.errors_impl.UnknownError: Could not start gRPC server [worker-0]: 2023-04-21 03:49:05.999051: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 18188289714339769995 [worker-2]: 2023-04-21 03:49:06.006029: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: 2023-04-21 03:49:06.136905: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 583364318242311187 [worker-1]: 2023-04-21 03:49:06.137890: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0421 03:49:06.147497 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0421 03:49:06.141692 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0421 03:49:06.148394 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0421 03:49:06.166907 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0421 03:49:06.209089 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0421 03:49:06.209676 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18979', 'localhost:19449', 'localhost:19921', 'localhost:24611']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0421 03:49:06.209864 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18979', 'localhost:19449', 'localhost:19921', 'localhost:24611']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0421 03:49:06.269159 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0421 03:49:06.269761 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18979', 'localhost:19449', 'localhost:19921', 'localhost:24611']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0421 03:49:06.269960 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18979', 'localhost:19449', 'localhost:19921', 'localhost:24611']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0421 03:49:06.338829 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0421 03:49:06.339427 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18979', 'localhost:19449', 'localhost:19921', 'localhost:24611']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0421 03:49:06.339619 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18979', 'localhost:19449', 'localhost:19921', 'localhost:24611']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0421 03:49:06.376888 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0421 03:49:06.377495 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18979', 'localhost:19449', 'localhost:19921', 'localhost:24611']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0421 03:49:06.377693 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18979', 'localhost:19449', 'localhost:19921', 'localhost:24611']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0421 03:49:06.582798 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-0]: I0421 03:49:06.636672 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0421 03:49:06.641704 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0421 03:49:06.643962 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-2]: I0421 03:49:06.629271 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-0]: I0421 03:49:06.644128 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: I0421 03:49:06.644640 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: I0421 03:49:06.643358 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0421 03:49:06.632639 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-1]: I0421 03:49:06.633828 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0421 03:49:06.644692 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0421 03:49:06.644561 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: I0421 03:49:06.644624 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: W0421 03:49:06.644983 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: I0421 03:49:06.644740 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: INFO:tensorflow:Start training at 0 [worker-2]: W0421 03:49:06.645009 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: INFO:tensorflow:['workertemp_1', 'workertemp_2', 'workertemp_3'] [worker-3]: I0421 03:49:06.645159 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: Instructions for updating: [worker-3]: INFO:tensorflow:['workertemp_1', 'workertemp_2', 'workertemp_3'] [worker-0]: I0421 03:49:06.651458 281472986018656 gce_failure_handler_test.py:203] ['workertemp_1', 'workertemp_2', 'workertemp_3'] [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: I0421 03:49:06.656421 281472986018656 gce_failure_handler_test.py:203] ['workertemp_1', 'workertemp_2', 'workertemp_3'] [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0421 03:49:06.645183 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: INFO:tensorflow:['workertemp_1', 'workertemp_2', 'workertemp_3'] [worker-1]: Instructions for updating: [worker-2]: I0421 03:49:06.684320 281472986018656 gce_failure_handler_test.py:203] ['workertemp_1', 'workertemp_2', 'workertemp_3'] [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0421 03:49:06.645178 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0421 03:49:06.645363 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: INFO:tensorflow:['workertemp_1', 'workertemp_2', 'workertemp_3'] [worker-1]: I0421 03:49:06.689286 281472986018656 gce_failure_handler_test.py:203] ['workertemp_1', 'workertemp_2', 'workertemp_3'] [worker-0]: Process _Process-6: [worker-0]: Traceback (most recent call last): [worker-0]: File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap [worker-0]: self.run() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 755, in _run_with_setenv [worker-0]: return self._actual_run() [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl [worker-0]: app.run(lambda _: self._run_impl()) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run [worker-0]: _run_main(main, args) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main [worker-0]: sys.exit(main(argv)) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in [worker-0]: app.run(lambda _: self._run_impl()) [worker-0]: File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 866, in __call__ [worker-0]: six.reraise(*info.exc_info) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise [worker-0]: raise value [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained [worker-0]: return_value = fn(*args, **kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 211, in worker_fn [worker-0]: self.assertNotEmpty(checkpoint_index) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 972, in assertNotEmpty [worker-0]: self.fail('{!r} has length of 0.'.format(container), msg) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 1814, in fail [worker-2]: Process _Process-4: [worker-2]: Traceback (most recent call last): [worker-2]: File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap [worker-2]: self.run() [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 755, in _run_with_setenv [worker-1]: Process _Process-3: [worker-2]: return self._actual_run() [worker-0]: return super(TestCase, self).fail(self._formatMessage(prefix, msg)) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl [worker-0]: File "/usr/lib/python3.8/unittest/case.py", line 753, in fail [worker-2]: app.run(lambda _: self._run_impl()) [worker-3]: Process _Process-5: [worker-0]: raise self.failureException(msg) [worker-3]: Traceback (most recent call last): [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run [worker-0]: AssertionError: [] has length of 0. [worker-3]: File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap [worker-2]: _run_main(main, args) [worker-3]: self.run() [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 755, in _run_with_setenv [worker-2]: sys.exit(main(argv)) [worker-3]: return self._actual_run() [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl [worker-2]: app.run(lambda _: self._run_impl()) [worker-3]: app.run(lambda _: self._run_impl()) [worker-2]: File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run [worker-2]: self._target(*self._args, **self._kwargs) [worker-3]: _run_main(main, args) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 866, in __call__ [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main [worker-2]: six.reraise(*info.exc_info) [worker-3]: sys.exit(main(argv)) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in [worker-2]: raise value [worker-3]: app.run(lambda _: self._run_impl()) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained [worker-3]: File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run [worker-2]: return_value = fn(*args, **kwargs) [worker-3]: self._target(*self._args, **self._kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 211, in worker_fn [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 866, in __call__ [worker-2]: self.assertNotEmpty(checkpoint_index) [worker-3]: six.reraise(*info.exc_info) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 972, in assertNotEmpty [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise [worker-2]: self.fail('{!r} has length of 0.'.format(container), msg) [worker-3]: raise value [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 1814, in fail [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained [worker-2]: return super(TestCase, self).fail(self._formatMessage(prefix, msg)) [worker-3]: return_value = fn(*args, **kwargs) [worker-2]: File "/usr/lib/python3.8/unittest/case.py", line 753, in fail [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 211, in worker_fn [worker-2]: raise self.failureException(msg) [worker-3]: self.assertNotEmpty(checkpoint_index) [worker-2]: AssertionError: [] has length of 0. [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 972, in assertNotEmpty [worker-3]: self.fail('{!r} has length of 0.'.format(container), msg) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 1814, in fail [worker-3]: return super(TestCase, self).fail(self._formatMessage(prefix, msg)) [worker-3]: File "/usr/lib/python3.8/unittest/case.py", line 753, in fail [worker-3]: raise self.failureException(msg) [worker-3]: AssertionError: [] has length of 0. [worker-1]: Traceback (most recent call last): [worker-1]: File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap [worker-1]: self.run() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 755, in _run_with_setenv [worker-1]: return self._actual_run() [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in _run_with_absl [worker-1]: app.run(lambda _: self._run_impl()) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 312, in run [worker-1]: _run_main(main, args) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/app.py", line 258, in _run_main [worker-1]: sys.exit(main(argv)) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_lib.py", line 54, in [worker-1]: app.run(lambda _: self._run_impl()) [worker-1]: File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 866, in __call__ [worker-1]: six.reraise(*info.exc_info) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise [worker-1]: raise value [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained [worker-1]: return_value = fn(*args, **kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 211, in worker_fn [worker-1]: self.assertNotEmpty(checkpoint_index) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 972, in assertNotEmpty [worker-1]: self.fail('{!r} has length of 0.'.format(container), msg) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 1814, in fail [worker-1]: return super(TestCase, self).fail(self._formatMessage(prefix, msg)) [worker-1]: File "/usr/lib/python3.8/unittest/case.py", line 753, in fail [worker-1]: raise self.failureException(msg) [worker-1]: AssertionError: [] has length of 0. I0421 03:49:08.056818 281472889680736 multi_process_runner.py:646] worker-0 exit code: 1 I0421 03:49:08.057142 281472889680736 multi_process_runner.py:646] worker-1 exit code: 1 I0421 03:49:08.057274 281472889680736 multi_process_runner.py:646] worker-2 exit code: 1 I0421 03:49:08.057403 281472889680736 multi_process_runner.py:646] worker-3 exit code: 1 INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_basic_run_test_inputarg_checkpoint_strategyoption_MWMSmultiworker): 36.1s I0421 03:49:08.171328 281472889680736 test_util.py:2462] time(__main__.GceFailureHandlingTest.test_basic_run_test_inputarg_checkpoint_strategyoption_MWMSmultiworker): 36.1s [ FAILED ] GceFailureHandlingTest.test_basic_run_test_inputarg_checkpoint_strategyoption_MWMSmultiworker [ RUN ] GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',) I0421 03:49:08.377237 281472889680736 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/device:CPU:0',) INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO I0421 03:49:08.377898 281472889680736 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO INFO:tensorflow:Start polling for termination signal. I0421 03:49:08.427135 281472889680736 failure_handling.py:683] Start polling for termination signal. INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. I0421 03:49:08.457927 281472889680736 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. W0421 03:49:08.458477 281472889680736 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. Instructions for updating: Track steps using a tf.Variable saved in checkpoint instead. INFO:tensorflow:Start training at 0 I0421 03:49:08.458669 281472889680736 gce_failure_handler_test.py:194] Start training at 0 WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff64f9c040> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W0421 03:49:08.798328 281472889680736 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff64f9c040> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff64f9c820> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. W0421 03:49:08.856400 281472889680736 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff64f9c820> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. INFO:tensorflow:epoch 0 finished I0421 03:49:08.856892 281472889680736 gce_failure_handler_test.py:192] epoch 0 finished INFO:tensorflow:epoch 1 finished I0421 03:49:09.218824 281472889680736 gce_failure_handler_test.py:192] epoch 1 finished INFO:tensorflow:epoch 2 finished I0421 03:49:09.425783 281472889680736 gce_failure_handler_test.py:192] epoch 2 finished INFO:tensorflow:epoch 3 finished I0421 03:49:09.706388 281472889680736 gce_failure_handler_test.py:192] epoch 3 finished INFO:tensorflow:epoch 4 finished I0421 03:49:09.907932 281472889680736 gce_failure_handler_test.py:192] epoch 4 finished INFO:tensorflow:Training finished. I0421 03:49:09.908371 281472889680736 gce_failure_handler_test.py:244] Training finished. INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker): 1.74s I0421 03:49:09.917257 281472889680736 test_util.py:2462] time(__main__.GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker): 1.74s [ OK ] GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_False_inputarg_checkpoint_strategyoption_MWMSmultiworker [ RUN ] GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker INFO:tensorflow:Using MirroredStrategy with devices ('/device:CPU:0',) I0421 03:49:09.928894 281472889680736 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/device:CPU:0',) INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO I0421 03:49:09.929369 281472889680736 collective_all_reduce_strategy.py:446] Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.AUTO INFO:tensorflow:Start polling for termination signal. I0421 03:49:09.946329 281472889680736 failure_handling.py:683] Start polling for termination signal. INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. I0421 03:49:09.966291 281472889680736 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. INFO:tensorflow:Start training at 0 I0421 03:49:09.966730 281472889680736 gce_failure_handler_test.py:194] Start training at 0 INFO:tensorflow:epoch 0 finished I0421 03:49:10.165224 281472889680736 gce_failure_handler_test.py:192] epoch 0 finished INFO:tensorflow:epoch 1 finished I0421 03:49:10.333051 281472889680736 gce_failure_handler_test.py:192] epoch 1 finished INFO:tensorflow:epoch 2 finished I0421 03:49:10.524244 281472889680736 gce_failure_handler_test.py:192] epoch 2 finished INFO:tensorflow:epoch 3 finished I0421 03:49:10.718724 281472889680736 gce_failure_handler_test.py:192] epoch 3 finished INFO:tensorflow:epoch 4 finished I0421 03:49:10.916886 281472889680736 gce_failure_handler_test.py:192] epoch 4 finished INFO:tensorflow:Training finished. I0421 03:49:10.917334 281472889680736 gce_failure_handler_test.py:244] Training finished. INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker): 1.0s I0421 03:49:10.923181 281472889680736 test_util.py:2462] time(__main__.GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker): 1.0s [ OK ] GceFailureHandlingTest.test_grace_period_continue_training_test_apiwrappingtrain_True_inputarg_checkpoint_strategyoption_MWMSmultiworker [ RUN ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_0_inputarg_checkpoint_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 23958 I0421 03:49:10.928553 281472889680736 test_util.py:3794] Using local port 23958 INFO:tensorflow:Using local port 18523 I0421 03:49:10.929147 281472889680736 test_util.py:3794] Using local port 18523 INFO:tensorflow:Using local port 16026 I0421 03:49:10.929542 281472889680736 test_util.py:3794] Using local port 16026 INFO:tensorflow:Using local port 16567 I0421 03:49:10.929920 281472889680736 test_util.py:3794] Using local port 16567 INFO:tensorflow:Cluster starting. I0421 03:49:10.947344 281472889680736 gce_failure_handler_test.py:405] Cluster starting. [worker-0]: I0421 03:49:10.989327 281472986018656 multi_process_runner.py:840] Subprocess with PID 1954435 (worker, 0) is now being started. [worker-0]: I0421 03:49:10.989763 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:23958", "localhost:18523", "localhost:16026", "localhost:16567"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-0]: 2023-04-21 03:49:11.207187: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:23958 [worker-0]: 2023-04-21 03:49:11.232514: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 17554395050177224991 [worker-0]: 2023-04-21 03:49:11.233234: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-1]: I0421 03:49:11.289668 281472986018656 multi_process_runner.py:840] Subprocess with PID 1954445 (worker, 1) is now being started. [worker-2]: I0421 03:49:11.299464 281472986018656 multi_process_runner.py:840] Subprocess with PID 1954636 (worker, 2) is now being started. [worker-1]: I0421 03:49:11.290097 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:23958", "localhost:18523", "localhost:16026", "localhost:16567"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I0421 03:49:11.299896 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:23958", "localhost:18523", "localhost:16026", "localhost:16567"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0421 03:49:11.330856 281472986018656 multi_process_runner.py:840] Subprocess with PID 1954728 (worker, 3) is now being started. [worker-3]: I0421 03:49:11.331291 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:23958", "localhost:18523", "localhost:16026", "localhost:16567"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-1]: 2023-04-21 03:49:11.407610: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:18523 [worker-0]: 2023-04-21 03:49:11.423828: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 14450499917649407908 [worker-1]: 2023-04-21 03:49:11.424053: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-2]: 2023-04-21 03:49:11.447141: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:16026 [worker-0]: 2023-04-21 03:49:11.460227: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 3807627181524018641 [worker-2]: 2023-04-21 03:49:11.461384: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-3]: 2023-04-21 03:49:11.567366: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:16567 [worker-0]: 2023-04-21 03:49:11.576203: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 7169585604704379309 [worker-3]: 2023-04-21 03:49:11.576708: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0421 03:49:11.578625 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0421 03:49:11.579286 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0421 03:49:11.582391 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0421 03:49:11.579280 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-0]: I0421 03:49:11.647020 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-2]: I0421 03:49:11.646999 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0421 03:49:11.647595 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23958', 'localhost:18523', 'localhost:16026', 'localhost:16567']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0421 03:49:11.647788 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23958', 'localhost:18523', 'localhost:16026', 'localhost:16567']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0421 03:49:11.647457 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23958', 'localhost:18523', 'localhost:16026', 'localhost:16567']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0421 03:49:11.647626 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23958', 'localhost:18523', 'localhost:16026', 'localhost:16567']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0421 03:49:11.652954 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0421 03:49:11.653551 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23958', 'localhost:18523', 'localhost:16026', 'localhost:16567']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0421 03:49:11.653738 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23958', 'localhost:18523', 'localhost:16026', 'localhost:16567']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0421 03:49:11.668564 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0421 03:49:11.668996 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23958', 'localhost:18523', 'localhost:16026', 'localhost:16567']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0421 03:49:11.669165 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23958', 'localhost:18523', 'localhost:16026', 'localhost:16567']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0421 03:49:11.787064 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-0]: I0421 03:49:11.788424 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-0]: Traceback (most recent call last): [worker-0]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0421 03:49:11.788896 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0421 03:49:11.789239 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0421 03:49:11.789418 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: self.run() [worker-0]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: if self._termination_watcher_fn(): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0421 03:49:11.825144 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-1]: I0421 03:49:11.826071 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-1]: Traceback (most recent call last): [worker-1]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-1]: self.run() [worker-1]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: if self._termination_watcher_fn(): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0421 03:49:11.828575 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0421 03:49:11.829005 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0421 03:49:11.829187 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0421 03:49:11.828835 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: I0421 03:49:11.838299 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-3]: I0421 03:49:11.856315 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0421 03:49:11.857186 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-3]: Traceback (most recent call last): [worker-3]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-3]: self.run() [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-2]: Traceback (most recent call last): [worker-3]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-2]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-3]: self._target(*self._args, **self._kwargs) [worker-2]: self.run() [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-3]: if self._termination_watcher_fn(): [worker-2]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: if self._termination_watcher_fn(): [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0421 03:49:11.868689 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0421 03:49:11.869263 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0421 03:49:11.870382 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0421 03:49:11.870913 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-2]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0421 03:49:11.869459 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0421 03:49:11.871101 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:11.923637 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:11.964160 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:11.979696 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:11.996672 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:12.116910 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:12.150398 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:12.170151 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:12.170128 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:12.243545 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:12.246454 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:12.244883 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:12.255847 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:12.340167 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:12.350409 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:12.357967 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:12.366232 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:12.504922 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:12.497096 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:12.525434 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:12.521562 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785cc0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785ce0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0421 03:49:12.875598 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785ce0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0421 03:49:12.869272 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785cc0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785cd0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0421 03:49:12.881590 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785cd0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785e70d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0421 03:49:12.886364 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785e70d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:12.892716 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:12.900184 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:12.910546 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:12.909895 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785e7940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0421 03:49:13.095790 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785e7940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I0421 03:49:13.096166 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785cc940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0421 03:49:13.102111 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785cc940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I0421 03:49:13.103088 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:13.103901 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:13.111688 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785ce940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0421 03:49:13.116367 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785ce940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:epoch 0 finished [worker-1]: I0421 03:49:13.116753 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785cd940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0421 03:49:13.116450 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785cd940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0421 03:49:13.116944 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:13.125185 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:13.128233 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:13.228420 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:13.252045 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:13.281261 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:13.282950 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:13.400730 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:13.440276 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:13.442039 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:13.446066 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:13.553388 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:13.582490 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:13.569219 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:13.591774 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:13.658332 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:13.674313 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:13.666256 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:13.677592 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:13.745839 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:13.747681 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:13.755787 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:13.756419 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 1 finished [worker-1]: I0421 03:49:13.802978 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0421 03:49:13.802787 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I0421 03:49:13.802442 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-2]: I0421 03:49:13.803387 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:13.811090 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:13.812997 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:13.818326 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:13.813014 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:13.929527 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:13.946684 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:13.963997 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:13.952728 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:14.045684 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:14.048303 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:14.049406 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:14.081033 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:14.146154 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:14.146107 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:14.146406 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:14.146392 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:14.209695 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:14.214858 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:14.228296 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:14.249074 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:14.329378 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:14.335562 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:14.348609 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:14.348333 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I0421 03:49:14.417233 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0421 03:49:14.428126 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0421 03:49:14.436467 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:14.437359 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-3]: I0421 03:49:14.427368 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:14.435079 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:14.443780 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:14.445757 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:14.512315 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:14.512379 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:14.513019 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:14.512681 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:14.577125 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:14.585990 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:14.607248 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:14.615596 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:14.700851 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:14.708908 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:14.721063 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:14.747437 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:14.848189 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:14.846623 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:14.854450 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:14.880580 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:14.948547 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:14.960521 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:14.960640 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:14.970494 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-3]: I0421 03:49:15.056841 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: I0421 03:49:15.057121 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: I0421 03:49:15.057349 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-2]: I0421 03:49:15.062737 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:15.072527 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:15.070367 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:15.080314 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:15.080093 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:15.220799 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:15.218185 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:15.234887 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:15.254191 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:15.477059 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:15.492275 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:15.497252 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:15.362242 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:15.584743 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:15.600872 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:15.600916 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:15.630969 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:15.749488 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:15.754592 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:15.760614 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:15.780698 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:15.854168 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:15.854317 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:15.858569 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:15.866766 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-3]: I0421 03:49:15.916254 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: INFO:tensorflow:Training finished. [worker-2]: INFO:tensorflow:epoch 4 finished [worker-1]: INFO:tensorflow:epoch 4 finished [worker-2]: I0421 03:49:15.916735 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: I0421 03:49:15.916352 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: I0421 03:49:15.916742 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: INFO:tensorflow:Training finished. [worker-2]: INFO:tensorflow:Training finished. [worker-3]: I0421 03:49:15.916536 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-1]: INFO:tensorflow:Training finished. [worker-2]: I0421 03:49:15.917155 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-0]: I0421 03:49:15.916633 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-1]: I0421 03:49:15.917157 281472986018656 gce_failure_handler_test.py:244] Training finished. INFO:tensorflow:restarting workers I0421 03:49:17.070843 281472889680736 gce_failure_handler_test.py:411] restarting workers INFO:tensorflow:workers restarted I0421 03:49:17.110833 281472889680736 gce_failure_handler_test.py:415] workers restarted [worker-0]: I0421 03:49:17.221368 281472986018656 multi_process_runner.py:840] Subprocess with PID 1965245 (worker, 0) is now being started. [worker-0]: I0421 03:49:17.221812 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:23958", "localhost:18523", "localhost:16026", "localhost:16567"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-2]: I0421 03:49:17.286773 281472986018656 multi_process_runner.py:840] Subprocess with PID 1965251 (worker, 2) is now being started. [worker-1]: I0421 03:49:17.290307 281472986018656 multi_process_runner.py:840] Subprocess with PID 1965248 (worker, 1) is now being started. [worker-3]: I0421 03:49:17.287966 281472986018656 multi_process_runner.py:840] Subprocess with PID 1965311 (worker, 3) is now being started. [worker-3]: I0421 03:49:17.288392 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:23958", "localhost:18523", "localhost:16026", "localhost:16567"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-2]: I0421 03:49:17.287240 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:23958", "localhost:18523", "localhost:16026", "localhost:16567"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-1]: I0421 03:49:17.290731 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:23958", "localhost:18523", "localhost:16026", "localhost:16567"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: 2023-04-21 03:49:17.376959: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:23958 [worker-0]: 2023-04-21 03:49:17.379932: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 11206528709619867648 [worker-0]: 2023-04-21 03:49:17.380151: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-1]: 2023-04-21 03:49:17.394420: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:18523 [worker-0]: 2023-04-21 03:49:17.401384: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 12066686607095024670 [worker-1]: 2023-04-21 03:49:17.401700: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-2]: 2023-04-21 03:49:17.410208: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:16026 [worker-3]: 2023-04-21 03:49:17.414780: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:16567 [worker-0]: 2023-04-21 03:49:17.429309: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 2734142721916096958 [worker-2]: 2023-04-21 03:49:17.429663: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: 2023-04-21 03:49:17.439531: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 13931656003840075461 [worker-3]: 2023-04-21 03:49:17.439737: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0421 03:49:17.441419 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I0421 03:49:17.441547 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: I0421 03:49:17.441537 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0421 03:49:17.441724 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0421 03:49:17.512370 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0421 03:49:17.512834 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23958', 'localhost:18523', 'localhost:16026', 'localhost:16567']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0421 03:49:17.513016 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23958', 'localhost:18523', 'localhost:16026', 'localhost:16567']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-0]: I0421 03:49:17.512393 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0421 03:49:17.512832 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: I0421 03:49:17.522791 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23958', 'localhost:18523', 'localhost:16026', 'localhost:16567']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0421 03:49:17.523249 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: I0421 03:49:17.513015 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23958', 'localhost:18523', 'localhost:16026', 'localhost:16567']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23958', 'localhost:18523', 'localhost:16026', 'localhost:16567']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0421 03:49:17.523429 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23958', 'localhost:18523', 'localhost:16026', 'localhost:16567']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0421 03:49:17.531418 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0421 03:49:17.531873 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23958', 'localhost:18523', 'localhost:16026', 'localhost:16567']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0421 03:49:17.532056 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:23958', 'localhost:18523', 'localhost:16026', 'localhost:16567']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0421 03:49:17.576734 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-0]: I0421 03:49:17.577534 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-0]: Traceback (most recent call last): [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-1]: I0421 03:49:17.578293 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: I0421 03:49:17.578313 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-0]: I0421 03:49:17.578083 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: I0421 03:49:17.579106 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: I0421 03:49:17.579080 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-0]: Instructions for updating: [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: Traceback (most recent call last): [worker-0]: W0421 03:49:17.578388 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-3]: Traceback (most recent call last): [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: Instructions for updating: [worker-3]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-2]: I0421 03:49:17.583058 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0421 03:49:17.579537 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: INFO:tensorflow:Start training at 0 [worker-1]: I0421 03:49:17.579800 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: I0421 03:49:17.584031 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-0]: I0421 03:49:17.578562 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: self.run() [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-3]: Instructions for updating: [worker-0]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-2]: Traceback (most recent call last): [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: self._target(*self._args, **self._kwargs) [worker-3]: W0421 03:49:17.579837 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-3]: Instructions for updating: [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: if self._termination_watcher_fn(): [worker-2]: I0421 03:49:17.584509 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: INFO:tensorflow:Start training at 0 [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0421 03:49:17.580177 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-2]: Instructions for updating: [worker-3]: I0421 03:49:17.580024 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: self.run() [worker-3]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-3]: if self._termination_watcher_fn(): [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: I0421 03:49:17.580375 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: self.run() [worker-2]: W0421 03:49:17.585030 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: Instructions for updating: [worker-1]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: INFO:tensorflow:Start training at 0 [worker-1]: if self._termination_watcher_fn(): [worker-2]: I0421 03:49:17.585389 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: self.run() [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: self._target(*self._args, **self._kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: if self._termination_watcher_fn(): [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:17.733813 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:17.734032 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:17.734491 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:17.849041 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:17.941480 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:17.948267 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:17.942029 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:17.957104 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:18.031890 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:18.070563 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:18.061062 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:18.092544 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:18.260828 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:18.251631 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:18.261051 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:18.246423 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:18.336859 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:18.348113 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:18.357187 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:18.383966 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785e70d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0421 03:49:18.448370 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785e70d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785ec0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0421 03:49:18.457330 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785ec0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785d10d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0421 03:49:18.466681 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785d10d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785eb0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0421 03:49:18.470053 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785eb0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:18.491054 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:18.510409 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:18.489991 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:18.570782 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785e7940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0421 03:49:18.666295 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785e7940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I0421 03:49:18.666662 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785d1940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0421 03:49:18.676667 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785d1940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I0421 03:49:18.677227 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785eb940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:18.690351 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: W0421 03:49:18.696669 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785eb940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:epoch 0 finished [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785ec940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0421 03:49:18.728747 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785ec940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0421 03:49:18.729138 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:18.722142 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:18.737493 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:18.697230 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:18.722773 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:18.834181 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:18.852524 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:18.839246 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:18.861804 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:18.980149 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:18.991650 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:18.986571 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:18.991628 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:19.075932 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:19.103044 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:19.111608 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:19.134967 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:19.253782 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:19.254522 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:19.246854 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:19.256317 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:19.341766 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:19.347013 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:19.347186 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:19.355618 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I0421 03:49:19.425911 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-1]: I0421 03:49:19.443657 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: I0421 03:49:19.439616 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:19.432408 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: I0421 03:49:19.451459 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:19.446484 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:19.471571 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 INFO:tensorflow:Termination notice available. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 I0421 03:49:19.496280 281469215961568 gce_failure_handler_test.py:142] Termination notice available. [worker-2]: I0421 03:49:19.482319 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 --- Logging error --- Traceback (most recent call last): File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit msg = self.format(record) File "/usr/lib/python3.8/logging/__init__.py", line 929, in format return fmt.format(record) File "/usr/lib/python3.8/logging/__init__.py", line 668, in format record.message = record.getMessage() File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage msg = msg % self.args TypeError: not all arguments converted during string formatting Call stack: File "/usr/lib/python3.8/threading.py", line 890, in _bootstrap self._bootstrap_inner() File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/usr/lib/python3.8/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 696, in _poll_termination_signal self._maybe_set_received_own_sigterm() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 701, in _maybe_set_received_own_sigterm logging.info('Received termination notice.', File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/platform/tf_logging.py", line 204, in info get_logger().info(msg, *args, **kwargs) Message: 'Received termination notice.' Arguments: ('single_worker',) --- Logging error --- Traceback (most recent call last): File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit msg = self.format(record) File "/usr/lib/python3.8/logging/__init__.py", line 929, in format return fmt.format(record) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/logging/__init__.py", line 1025, in format return prefix + super(PythonFormatter, self).format(record) File "/usr/lib/python3.8/logging/__init__.py", line 668, in format record.message = record.getMessage() File "/usr/lib/python3.8/logging/__init__.py", line 373, in getMessage msg = msg % self.args TypeError: not all arguments converted during string formatting Call stack: File "/usr/lib/python3.8/threading.py", line 890, in _bootstrap self._bootstrap_inner() File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/usr/lib/python3.8/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 696, in _poll_termination_signal self._maybe_set_received_own_sigterm() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 701, in _maybe_set_received_own_sigterm logging.info('Received termination notice.', File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/platform/tf_logging.py", line 204, in info get_logger().info(msg, *args, **kwargs) File "/usr/lib/python3.8/logging/__init__.py", line 1446, in info self._log(INFO, msg, args, **kwargs) File "/usr/lib/python3.8/logging/__init__.py", line 1589, in _log self.handle(record) File "/usr/lib/python3.8/logging/__init__.py", line 1599, in handle self.callHandlers(record) File "/usr/lib/python3.8/logging/__init__.py", line 1661, in callHandlers hdlr.handle(record) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/logging/__init__.py", line 988, in handle return self._current_handler.handle(record) File "/usr/lib/python3.8/logging/__init__.py", line 954, in handle self.emit(record) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/logging/__init__.py", line 925, in emit super(PythonHandler, self).emit(record) Message: 'Received termination notice.' Arguments: ('single_worker',) Exception ignored in: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 775, in __del__ self._stop_poll_termination_signal_thread() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 734, in _stop_poll_termination_signal_thread self._poll_termination_signal_thread.join() File "/usr/lib/python3.8/threading.py", line 1008, in join raise RuntimeError("cannot join current thread") RuntimeError: cannot join current thread [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:19.574203 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:19.559748 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:19.574469 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:19.594172 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:19.655451 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:19.654478 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:19.657699 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:19.660719 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:19.723621 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:19.727078 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:19.727115 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:19.741743 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:19.813527 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:19.807711 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:19.836842 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:19.841928 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:19.907434 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:19.920743 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:19.941944 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:19.937951 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I0421 03:49:20.013927 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0421 03:49:20.022740 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: I0421 03:49:20.022737 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-0]: I0421 03:49:20.028995 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:20.048609 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:20.041899 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:20.051912 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:20.056762 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:20.191802 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:20.201784 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:20.232323 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:20.218665 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:20.372173 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:20.362554 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:20.381478 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:20.392182 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:20.467200 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:20.467360 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:20.492475 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:20.516248 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:20.610827 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:20.614547 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:20.612325 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:20.622192 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:20.724103 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:20.723155 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:20.741586 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:20.761955 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-3]: I0421 03:49:20.849328 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-1]: I0421 03:49:20.865210 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: I0421 03:49:20.858046 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-2]: I0421 03:49:20.878353 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:20.872391 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:20.882331 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:20.898095 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:20.932187 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:21.074371 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:21.081115 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:21.084471 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:21.084537 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:21.184842 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:21.184750 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:21.334407 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:21.363432 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:21.465530 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:21.465332 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:21.477743 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:21.491858 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:21.602526 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:21.604136 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:21.612576 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:21.632957 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:21.739757 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:21.745285 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:21.751168 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:21.757793 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-3]: I0421 03:49:21.837039 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: INFO:tensorflow:epoch 4 finished [worker-3]: INFO:tensorflow:Training finished. [worker-2]: I0421 03:49:21.837617 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: I0421 03:49:21.837471 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-2]: INFO:tensorflow:Training finished. [worker-2]: I0421 03:49:21.837908 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-1]: INFO:tensorflow:epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-1]: I0421 03:49:21.846663 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I0421 03:49:21.847092 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-0]: I0421 03:49:21.846462 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I0421 03:49:21.846756 281472986018656 gce_failure_handler_test.py:244] Training finished. I0421 03:49:22.090652 281472889680736 multi_process_runner.py:646] worker-0 exit code: 0 I0421 03:49:22.091016 281472889680736 multi_process_runner.py:646] worker-1 exit code: 0 I0421 03:49:22.091159 281472889680736 multi_process_runner.py:646] worker-2 exit code: 0 I0421 03:49:22.091284 281472889680736 multi_process_runner.py:646] worker-3 exit code: 0 I0421 03:49:22.094184 281472889680736 multi_process_runner.py:662] Joining log reading threads. I0421 03:49:22.094588 281472889680736 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_0_inputarg_checkpoint_strategyoption_MWMSmultiworker): 11.38s I0421 03:49:22.304374 281472889680736 test_util.py:2462] time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_0_inputarg_checkpoint_strategyoption_MWMSmultiworker): 11.38s [ OK ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_0_inputarg_checkpoint_strategyoption_MWMSmultiworker [ RUN ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_7_inputarg_checkpoint_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 18123 I0421 03:49:22.306369 281472889680736 test_util.py:3794] Using local port 18123 INFO:tensorflow:Using local port 22264 I0421 03:49:22.306898 281472889680736 test_util.py:3794] Using local port 22264 INFO:tensorflow:Using local port 24169 I0421 03:49:22.307292 281472889680736 test_util.py:3794] Using local port 24169 INFO:tensorflow:Using local port 23516 I0421 03:49:22.307666 281472889680736 test_util.py:3794] Using local port 23516 INFO:tensorflow:Cluster starting. I0421 03:49:22.340008 281472889680736 gce_failure_handler_test.py:405] Cluster starting. [worker-3]: I0421 03:49:22.439544 281472986018656 multi_process_runner.py:840] Subprocess with PID 1975201 (worker, 3) is now being started. [worker-0]: I0421 03:49:22.440593 281472986018656 multi_process_runner.py:840] Subprocess with PID 1975173 (worker, 0) is now being started. [worker-3]: I0421 03:49:22.439976 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:18123", "localhost:22264", "localhost:24169", "localhost:23516"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: I0421 03:49:22.441013 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:18123", "localhost:22264", "localhost:24169", "localhost:23516"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-0]: 2023-04-21 03:49:22.496511: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:18123 [worker-0]: 2023-04-21 03:49:22.536555: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 18394635766641585521 [worker-0]: 2023-04-21 03:49:22.537462: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-2]: I0421 03:49:22.539916 281472986018656 multi_process_runner.py:840] Subprocess with PID 1975188 (worker, 2) is now being started. [worker-2]: I0421 03:49:22.540361 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:18123", "localhost:22264", "localhost:24169", "localhost:23516"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-1]: I0421 03:49:22.556472 281472986018656 multi_process_runner.py:840] Subprocess with PID 1975181 (worker, 1) is now being started. [worker-1]: I0421 03:49:22.556902 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:18123", "localhost:22264", "localhost:24169", "localhost:23516"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-1]: 2023-04-21 03:49:22.617075: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:22264 [worker-0]: 2023-04-21 03:49:22.620896: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 3426890055639682351 [worker-1]: 2023-04-21 03:49:22.626440: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-3]: 2023-04-21 03:49:22.657224: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:23516 [worker-0]: 2023-04-21 03:49:22.668257: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 7121706329449934126 [worker-3]: 2023-04-21 03:49:22.668514: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-2]: 2023-04-21 03:49:22.746894: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:24169 [worker-0]: 2023-04-21 03:49:22.749605: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 17518651331620982219 [worker-2]: 2023-04-21 03:49:22.749798: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0421 03:49:22.751418 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0421 03:49:22.751636 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0421 03:49:22.757385 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0421 03:49:22.776076 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0421 03:49:22.821366 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0421 03:49:22.821807 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18123', 'localhost:22264', 'localhost:24169', 'localhost:23516']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0421 03:49:22.821978 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18123', 'localhost:22264', 'localhost:24169', 'localhost:23516']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0421 03:49:22.831401 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0421 03:49:22.831857 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18123', 'localhost:22264', 'localhost:24169', 'localhost:23516']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0421 03:49:22.832030 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18123', 'localhost:22264', 'localhost:24169', 'localhost:23516']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0421 03:49:22.857141 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0421 03:49:22.857567 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18123', 'localhost:22264', 'localhost:24169', 'localhost:23516']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0421 03:49:22.857738 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18123', 'localhost:22264', 'localhost:24169', 'localhost:23516']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0421 03:49:22.904261 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0421 03:49:22.904706 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18123', 'localhost:22264', 'localhost:24169', 'localhost:23516']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0421 03:49:22.904879 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18123', 'localhost:22264', 'localhost:24169', 'localhost:23516']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0421 03:49:23.080040 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0421 03:49:23.080747 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-0]: I0421 03:49:23.087182 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: Traceback (most recent call last): [worker-2]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0421 03:49:23.087559 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0421 03:49:23.088110 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0421 03:49:23.088480 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: self.run() [worker-2]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-2]: self._target(*self._args, **self._kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: if self._termination_watcher_fn(): [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0421 03:49:23.097489 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-1]: I0421 03:49:23.098436 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-1]: Traceback (most recent call last): [worker-1]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0421 03:49:23.099264 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: self.run() [worker-1]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: self._target(*self._args, **self._kwargs) [worker-3]: I0421 03:49:23.101010 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: if self._termination_watcher_fn(): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0421 03:49:23.100589 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0421 03:49:23.101254 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-3]: I0421 03:49:23.110483 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-3]: Traceback (most recent call last): [worker-3]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0421 03:49:23.111070 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0421 03:49:23.111530 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0421 03:49:23.111948 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: self.run() [worker-3]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-3]: if self._termination_watcher_fn(): [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-0]: I0421 03:49:23.116262 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-0]: Traceback (most recent call last): [worker-0]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0421 03:49:23.120631 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0421 03:49:23.121057 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0421 03:49:23.121223 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: self.run() [worker-0]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: if self._termination_watcher_fn(): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:23.284845 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:23.271947 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:23.276783 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:23.300644 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:23.427324 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:23.461913 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:23.479913 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:23.539856 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:23.644690 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:23.656890 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:23.709700 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:23.730355 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:23.789412 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:23.790112 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:23.790987 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:23.789364 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:23.905820 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:23.905877 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:23.911078 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:23.911179 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785e80d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0421 03:49:23.957076 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785e80d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785e70d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785eb0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0421 03:49:23.957373 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785eb0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0421 03:49:23.956966 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785e70d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:23.965215 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785ea0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0421 03:49:23.961800 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785ea0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:23.965507 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:23.987121 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:23.994869 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785ea940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0421 03:49:24.052056 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785ea940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0421 03:49:24.052440 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785e8940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0421 03:49:24.054618 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785e8940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I0421 03:49:24.054969 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785eb940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0421 03:49:24.059199 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785eb940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:epoch 0 finished [worker-1]: I0421 03:49:24.059548 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:24.067478 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785e7940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0421 03:49:24.066391 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785e7940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I0421 03:49:24.066753 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:24.080299 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:24.090062 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:24.102742 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:24.163987 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:24.169767 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:24.163357 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:24.181569 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:24.250978 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:24.252226 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:24.252233 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:24.253765 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:24.309785 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:24.311059 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:24.315227 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:24.316467 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:24.367811 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:24.370199 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:24.380396 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:24.389667 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:24.444408 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:24.455030 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:24.455051 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:24.467682 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I0421 03:49:24.518684 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-1]: I0421 03:49:24.527108 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:24.535587 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-2]: I0421 03:49:24.539597 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: I0421 03:49:24.539587 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: I0421 03:49:24.527224 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:24.548115 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:24.562056 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:24.646767 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:24.646671 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:24.653127 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:24.653421 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:24.735462 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:24.740037 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:24.733305 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:24.760336 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:24.821133 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:24.821172 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:24.825455 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:24.828543 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:24.915812 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:24.940317 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:24.940416 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:24.950284 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:25.033679 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:25.037248 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:25.027902 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:25.050463 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I0421 03:49:25.101071 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-0]: I0421 03:49:25.112431 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0421 03:49:25.113413 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0421 03:49:25.114721 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:25.122359 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:25.118475 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:25.123945 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:25.142068 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:25.251008 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:25.251799 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:25.254449 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:25.257786 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:25.329232 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:25.362130 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:25.365322 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:25.362226 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:25.488904 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:25.492146 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:25.495329 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:25.498105 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:25.581507 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:25.582647 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:25.584193 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:25.611818 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:25.695851 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:25.712877 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:25.722172 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:25.727166 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 3 finished [worker-0]: I0421 03:49:25.788641 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-3]: INFO:tensorflow:epoch 3 finished [worker-2]: I0421 03:49:25.795440 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: I0421 03:49:25.795259 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-3]: I0421 03:49:25.788524 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:25.807230 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:25.811679 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:25.832188 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:25.832115 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:25.932340 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:25.943982 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:25.949394 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:25.954045 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:26.066691 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:26.227296 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:26.257444 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:26.111513 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:26.364773 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:26.377100 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:26.391924 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:26.401609 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:26.478018 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:26.475294 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:26.487600 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:26.503191 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:26.570446 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:26.587692 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:26.608120 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:26.626466 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-1]: INFO:tensorflow:epoch 4 finished [worker-3]: I0421 03:49:26.709275 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: I0421 03:49:26.711291 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: INFO:tensorflow:Training finished. [worker-0]: I0421 03:49:26.709469 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: INFO:tensorflow:Training finished. [worker-3]: I0421 03:49:26.709676 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-1]: I0421 03:49:26.711657 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-0]: INFO:tensorflow:Training finished. [worker-2]: INFO:tensorflow:epoch 4 finished [worker-0]: I0421 03:49:26.709849 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-2]: I0421 03:49:26.740428 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: INFO:tensorflow:Training finished. [worker-2]: I0421 03:49:26.740830 281472986018656 gce_failure_handler_test.py:244] Training finished. INFO:tensorflow:restarting workers I0421 03:49:28.426692 281472889680736 gce_failure_handler_test.py:411] restarting workers INFO:tensorflow:workers restarted I0421 03:49:28.539711 281472889680736 gce_failure_handler_test.py:415] workers restarted [worker-0]: I0421 03:49:28.618716 281472986018656 multi_process_runner.py:840] Subprocess with PID 1983219 (worker, 0) is now being started. [worker-1]: I0421 03:49:28.634675 281472986018656 multi_process_runner.py:840] Subprocess with PID 1983222 (worker, 1) is now being started. [worker-2]: I0421 03:49:28.649335 281472986018656 multi_process_runner.py:840] Subprocess with PID 1983227 (worker, 2) is now being started. [worker-1]: I0421 03:49:28.635144 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:18123", "localhost:22264", "localhost:24169", "localhost:23516"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: I0421 03:49:28.619181 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:18123", "localhost:22264", "localhost:24169", "localhost:23516"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-2]: I0421 03:49:28.649778 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:18123", "localhost:22264", "localhost:24169", "localhost:23516"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0421 03:49:28.707502 281472986018656 multi_process_runner.py:840] Subprocess with PID 1983230 (worker, 3) is now being started. [worker-3]: I0421 03:49:28.707957 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:18123", "localhost:22264", "localhost:24169", "localhost:23516"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-1]: 2023-04-21 03:49:28.727300: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:22264 [worker-3]: 2023-04-21 03:49:28.743889: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:23516 [worker-0]: 2023-04-21 03:49:28.755886: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:18123 [worker-0]: 2023-04-21 03:49:28.767072: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 6776875820307531486 [worker-1]: 2023-04-21 03:49:28.774156: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-2]: 2023-04-21 03:49:28.787815: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:24169 [worker-0]: 2023-04-21 03:49:28.805205: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 10776479116161808984 [worker-2]: 2023-04-21 03:49:28.805695: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: 2023-04-21 03:49:28.807805: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 9947308916876696291 [worker-0]: 2023-04-21 03:49:28.807948: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: 2023-04-21 03:49:28.815274: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 501695030477779853 [worker-3]: 2023-04-21 03:49:28.817445: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0421 03:49:28.821279 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0421 03:49:28.822335 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0421 03:49:28.827334 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0421 03:49:28.833072 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0421 03:49:28.894915 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0421 03:49:28.895368 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18123', 'localhost:22264', 'localhost:24169', 'localhost:23516']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0421 03:49:28.895542 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18123', 'localhost:22264', 'localhost:24169', 'localhost:23516']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0421 03:49:28.897312 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0421 03:49:28.897725 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18123', 'localhost:22264', 'localhost:24169', 'localhost:23516']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0421 03:49:28.897892 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18123', 'localhost:22264', 'localhost:24169', 'localhost:23516']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0421 03:49:28.897315 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0421 03:49:28.897726 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18123', 'localhost:22264', 'localhost:24169', 'localhost:23516']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0421 03:49:28.897893 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18123', 'localhost:22264', 'localhost:24169', 'localhost:23516']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0421 03:49:28.907964 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0421 03:49:28.908381 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18123', 'localhost:22264', 'localhost:24169', 'localhost:23516']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0421 03:49:28.908551 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:18123', 'localhost:22264', 'localhost:24169', 'localhost:23516']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0421 03:49:28.963291 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0421 03:49:28.965104 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-2]: Traceback (most recent call last): [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0421 03:49:28.971138 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: I0421 03:49:28.971071 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-2]: self.run() [worker-2]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-2]: self._target(*self._args, **self._kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: if self._termination_watcher_fn(): [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0421 03:49:28.974710 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: I0421 03:49:28.968161 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-2]: Instructions for updating: [worker-0]: I0421 03:49:28.975843 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-2]: W0421 03:49:28.968495 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Traceback (most recent call last): [worker-2]: Instructions for updating: [worker-0]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: I0421 03:49:28.976450 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: INFO:tensorflow:Start training at 0 [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: I0421 03:49:28.968658 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0421 03:49:28.976751 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: self.run() [worker-0]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: if self._termination_watcher_fn(): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0421 03:49:28.978253 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-3]: I0421 03:49:28.986466 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-3]: Traceback (most recent call last): [worker-3]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-3]: self.run() [worker-3]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-3]: self._target(*self._args, **self._kwargs) [worker-1]: I0421 03:49:28.986685 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-1]: Traceback (most recent call last): [worker-1]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0421 03:49:28.987434 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0421 03:49:28.987813 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0421 03:49:28.988071 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: self.run() [worker-1]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: if self._termination_watcher_fn(): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-3]: if self._termination_watcher_fn(): [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0421 03:49:28.989395 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0421 03:49:28.989733 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0421 03:49:28.989895 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:29.094844 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:29.102890 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:29.118738 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:29.118767 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:29.259941 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:29.258754 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:29.263473 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:29.280147 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:29.360046 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:29.362775 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:29.368701 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:29.439803 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:29.539957 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:29.560078 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:29.552137 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:29.550323 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:29.686555 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:29.701276 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:29.700219 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:29.706963 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785d00d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785eb0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0421 03:49:29.806012 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785d00d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0421 03:49:29.806259 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785eb0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:29.815298 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785ec0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0421 03:49:29.817322 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785ec0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:29.830403 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785cf0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0421 03:49:29.822557 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785cf0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:29.849396 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:29.850345 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785d0940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785cf940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0421 03:49:29.943294 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785cf940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:epoch 0 finished [worker-1]: I0421 03:49:29.943637 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785eb940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0421 03:49:29.943105 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785eb940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0421 03:49:29.937225 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785d0940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I0421 03:49:29.937578 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I0421 03:49:29.943454 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:29.951747 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785ec940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0421 03:49:29.943543 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785ec940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:29.943894 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: I0421 03:49:29.973691 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:29.970398 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:30.000173 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:30.096533 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:30.105016 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:30.108799 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:30.120236 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:30.229663 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:30.234168 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:30.230600 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:30.250984 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:30.360158 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:30.367363 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:30.366666 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:30.381738 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:30.460271 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:30.470786 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:30.474146 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:30.491095 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:30.605322 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:30.618096 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:30.617572 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:30.600058 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I0421 03:49:30.687159 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0421 03:49:30.688991 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-2]: I0421 03:49:30.694461 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-1]: I0421 03:49:30.696479 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:30.704423 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:30.698477 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:30.720352 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:30.730276 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:30.871989 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:30.861420 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:30.866674 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:30.907060 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:30.996218 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:31.004810 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:31.000350 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:31.060400 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:31.160298 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:31.149106 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:31.160414 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:31.193602 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:31.254626 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:31.264447 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:31.280450 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:31.280803 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:31.345352 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:31.345063 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:31.351490 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:31.371789 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-3]: I0421 03:49:31.456582 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0421 03:49:31.467185 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0421 03:49:31.468242 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: I0421 03:49:31.457144 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:31.477956 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:31.478739 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:31.480433 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:31.480339 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:31.553425 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:31.553289 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:31.559323 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:31.561320 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:31.617108 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:31.627453 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:31.651797 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:31.669092 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:31.726255 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:31.726635 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:31.726800 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:31.751532 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:31.809559 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:31.810207 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:31.810903 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:31.827940 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:31.888791 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:31.905497 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:31.889720 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:31.895040 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 3 finished [worker-0]: I0421 03:49:31.977202 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-3]: INFO:tensorflow:epoch 3 finished [worker-3]: I0421 03:49:31.974713 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-2]: I0421 03:49:31.985008 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: I0421 03:49:31.985069 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:31.992166 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:31.986012 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:31.995022 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:31.999389 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:32.057785 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:32.057811 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:32.071139 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:32.071490 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:32.335919 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:32.340712 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:32.342161 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:32.219183 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:32.485622 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:32.486299 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:32.501827 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:32.537128 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:32.658525 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:32.671744 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:32.680950 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:32.700461 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:32.799066 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:32.821887 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:32.840647 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:32.851654 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-3]: I0421 03:49:32.951225 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I0421 03:49:32.951535 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-0]: INFO:tensorflow:epoch 4 finished [worker-2]: INFO:tensorflow:epoch 4 finished [worker-0]: I0421 03:49:32.954610 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: INFO:tensorflow:epoch 4 finished [worker-2]: I0421 03:49:32.955029 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: I0421 03:49:32.955103 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: INFO:tensorflow:Training finished. [worker-1]: INFO:tensorflow:Training finished. [worker-0]: INFO:tensorflow:Training finished. [worker-2]: I0421 03:49:32.955418 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-1]: I0421 03:49:32.955492 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-0]: I0421 03:49:32.955029 281472986018656 gce_failure_handler_test.py:244] Training finished. I0421 03:49:33.481168 281472889680736 multi_process_runner.py:646] worker-0 exit code: 0 I0421 03:49:33.481487 281472889680736 multi_process_runner.py:646] worker-1 exit code: 0 I0421 03:49:33.481623 281472889680736 multi_process_runner.py:646] worker-2 exit code: 0 I0421 03:49:33.481745 281472889680736 multi_process_runner.py:646] worker-3 exit code: 0 I0421 03:49:33.484345 281472889680736 multi_process_runner.py:662] Joining log reading threads. I0421 03:49:33.484543 281472889680736 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_7_inputarg_checkpoint_strategyoption_MWMSmultiworker): 11.37s I0421 03:49:33.675547 281472889680736 test_util.py:2462] time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_7_inputarg_checkpoint_strategyoption_MWMSmultiworker): 11.37s [ OK ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_False_graceperiod_7_inputarg_checkpoint_strategyoption_MWMSmultiworker [ RUN ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_0_inputarg_checkpoint_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 24816 I0421 03:49:33.677786 281472889680736 test_util.py:3794] Using local port 24816 INFO:tensorflow:Using local port 18415 I0421 03:49:33.678303 281472889680736 test_util.py:3794] Using local port 18415 INFO:tensorflow:Using local port 18121 I0421 03:49:33.678698 281472889680736 test_util.py:3794] Using local port 18121 INFO:tensorflow:Using local port 24939 I0421 03:49:33.679069 281472889680736 test_util.py:3794] Using local port 24939 INFO:tensorflow:Cluster starting. I0421 03:49:33.710082 281472889680736 gce_failure_handler_test.py:405] Cluster starting. [worker-0]: I0421 03:49:33.796802 281472986018656 multi_process_runner.py:840] Subprocess with PID 1990694 (worker, 0) is now being started. [worker-0]: I0421 03:49:33.797308 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24816", "localhost:18415", "localhost:18121", "localhost:24939"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I0421 03:49:33.975066 281472986018656 multi_process_runner.py:840] Subprocess with PID 1990934 (worker, 1) is now being started. [worker-0]: 2023-04-21 03:49:33.977677: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:24816 [worker-1]: I0421 03:49:33.975604 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24816", "localhost:18415", "localhost:18121", "localhost:24939"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I0421 03:49:33.991902 281472986018656 multi_process_runner.py:840] Subprocess with PID 1991060 (worker, 2) is now being started. [worker-3]: I0421 03:49:34.008535 281472986018656 multi_process_runner.py:840] Subprocess with PID 1991064 (worker, 3) is now being started. [worker-2]: I0421 03:49:33.992441 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24816", "localhost:18415", "localhost:18121", "localhost:24939"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-0]: 2023-04-21 03:49:34.016746: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 13730454206840944829 [worker-0]: 2023-04-21 03:49:34.020770: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-3]: I0421 03:49:34.009073 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24816", "localhost:18415", "localhost:18121", "localhost:24939"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-1]: 2023-04-21 03:49:34.081558: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:18415 [worker-0]: 2023-04-21 03:49:34.084831: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 16778179413555684471 [worker-1]: 2023-04-21 03:49:34.085120: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-2]: 2023-04-21 03:49:34.347000: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:18121 [worker-3]: 2023-04-21 03:49:34.362533: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:24939 [worker-0]: 2023-04-21 03:49:34.367707: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 16345003882801810500 [worker-2]: 2023-04-21 03:49:34.377062: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: 2023-04-21 03:49:34.377540: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 841329403421452977 [worker-3]: 2023-04-21 03:49:34.378083: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0421 03:49:34.385263 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0421 03:49:34.380176 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0421 03:49:34.397968 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0421 03:49:34.397720 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0421 03:49:34.462937 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0421 03:49:34.463549 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24816', 'localhost:18415', 'localhost:18121', 'localhost:24939']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0421 03:49:34.463742 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24816', 'localhost:18415', 'localhost:18121', 'localhost:24939']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0421 03:49:34.472601 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0421 03:49:34.473179 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24816', 'localhost:18415', 'localhost:18121', 'localhost:24939']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0421 03:49:34.473367 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24816', 'localhost:18415', 'localhost:18121', 'localhost:24939']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0421 03:49:34.481810 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0421 03:49:34.482339 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24816', 'localhost:18415', 'localhost:18121', 'localhost:24939']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0421 03:49:34.482523 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24816', 'localhost:18415', 'localhost:18121', 'localhost:24939']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0421 03:49:34.567126 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0421 03:49:34.567716 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24816', 'localhost:18415', 'localhost:18121', 'localhost:24939']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0421 03:49:34.567906 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24816', 'localhost:18415', 'localhost:18121', 'localhost:24939']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0421 03:49:34.651842 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0421 03:49:34.652925 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-3]: I0421 03:49:34.652922 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-0]: Traceback (most recent call last): [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0421 03:49:34.653864 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: I0421 03:49:34.654424 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-0]: I0421 03:49:34.653805 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-1]: I0421 03:49:34.654831 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-0]: self.run() [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-0]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Traceback (most recent call last): [worker-0]: Instructions for updating: [worker-1]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: self.run() [worker-0]: W0421 03:49:34.655996 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Traceback (most recent call last): [worker-0]: Instructions for updating: [worker-1]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: if self._termination_watcher_fn(): [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: I0421 03:49:34.671834 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0421 03:49:34.672627 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: Traceback (most recent call last): [worker-1]: I0421 03:49:34.656581 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: INFO:tensorflow:Start training at 0 [worker-3]: I0421 03:49:34.655132 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: I0421 03:49:34.656235 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: if self._termination_watcher_fn(): [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-3]: W0421 03:49:34.655451 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: Instructions for updating: [worker-3]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: self.run() [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0421 03:49:34.655612 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-3]: self.run() [worker-2]: self._target(*self._args, **self._kwargs) [worker-3]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-3]: self._target(*self._args, **self._kwargs) [worker-2]: if self._termination_watcher_fn(): [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: W0421 03:49:34.656958 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: Instructions for updating: [worker-3]: if self._termination_watcher_fn(): [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0421 03:49:34.657132 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: I0421 03:49:34.688485 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0421 03:49:34.688975 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0421 03:49:34.689154 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:34.766207 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:34.766529 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:34.766660 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:34.826092 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:34.902460 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:34.902877 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:34.908273 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:34.917431 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:34.981139 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:35.000651 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:35.000761 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:35.031203 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:35.104192 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:35.111310 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:35.104999 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:35.130805 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:35.185645 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:35.186443 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:35.185200 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:35.195074 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785cd0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0421 03:49:35.258899 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785cd0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785eb0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785ed0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0421 03:49:35.257547 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785eb0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785e40d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0421 03:49:35.258901 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785e40d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:35.268008 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: W0421 03:49:35.257556 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785ed0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:35.268053 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:35.268377 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:35.268950 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785ed940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785eb940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785e4940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785cd940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0421 03:49:35.315477 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785eb940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0421 03:49:35.315550 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785cd940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0421 03:49:35.315414 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785e4940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 0 finished [worker-2]: INFO:tensorflow:epoch 0 finished [worker-1]: INFO:tensorflow:epoch 0 finished [worker-0]: W0421 03:49:35.315295 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785ed940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: I0421 03:49:35.315948 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: I0421 03:49:35.316009 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: I0421 03:49:35.315893 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: INFO:tensorflow:epoch 0 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:35.315763 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: I0421 03:49:35.326833 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:35.324611 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:35.324581 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:35.388784 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:35.459017 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:35.459109 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:35.465218 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:35.467791 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:35.538683 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:35.538727 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:35.542696 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:35.543962 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:35.602098 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:35.601140 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:35.621584 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:35.621644 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:35.685859 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:35.695017 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:35.702672 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:35.710970 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:35.799589 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:35.801179 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:35.811281 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:35.807331 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I0421 03:49:35.870971 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0421 03:49:35.871195 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-2]: I0421 03:49:35.871375 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-1]: I0421 03:49:35.876382 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:35.881011 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:35.880811 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:35.901124 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:35.887287 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:36.015454 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:36.011026 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:36.008709 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:36.043244 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:36.104804 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:36.102330 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:36.111659 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:36.127261 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:36.194250 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:36.193528 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:36.193837 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:36.240606 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:36.304592 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:36.305140 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:36.329986 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:36.343198 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:36.437944 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:36.451383 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:36.450999 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:36.455270 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0421 03:49:36.529641 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-3]: INFO:tensorflow:epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-3]: I0421 03:49:36.529339 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: I0421 03:49:36.529896 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:36.540145 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:36.538363 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:36.539935 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:36.534657 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:36.560164 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:36.612554 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:36.637301 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:36.636231 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:36.650792 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:36.726217 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:36.741077 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:36.761311 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:36.740646 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:36.826077 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:36.829144 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:36.856331 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:36.860864 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:36.932553 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:36.945406 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:36.960604 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:36.951105 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:37.035360 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:37.036546 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:37.048191 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:37.058622 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-1]: I0421 03:49:37.112743 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: I0421 03:49:37.112803 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-3]: INFO:tensorflow:epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-3]: I0421 03:49:37.112209 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:37.121490 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:37.122654 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:37.112658 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:37.141153 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:37.160957 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:37.278642 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:37.281633 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:37.283798 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:37.287227 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:37.458979 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:37.345214 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:37.492262 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:37.360899 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:37.549007 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:37.549217 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:37.555302 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:37.550101 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:37.616345 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:37.616749 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:37.612298 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:37.631585 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:37.722913 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:37.721307 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:37.748424 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:37.772042 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-3]: I0421 03:49:37.834688 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: INFO:tensorflow:epoch 4 finished [worker-1]: I0421 03:49:37.836023 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: I0421 03:49:37.834420 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: INFO:tensorflow:Training finished. [worker-3]: INFO:tensorflow:Training finished. [worker-1]: I0421 03:49:37.836441 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-3]: I0421 03:49:37.835106 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I0421 03:49:37.834767 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I0421 03:49:37.846560 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: INFO:tensorflow:Training finished. [worker-2]: I0421 03:49:37.846976 281472986018656 gce_failure_handler_test.py:244] Training finished. INFO:tensorflow:restarting workers I0421 03:49:39.926724 281472889680736 gce_failure_handler_test.py:411] restarting workers [worker-0]: I0421 03:49:39.965538 281472986018656 multi_process_runner.py:840] Subprocess with PID 1995524 (worker, 0) is now being started. INFO:tensorflow:workers restarted I0421 03:49:39.976881 281472889680736 gce_failure_handler_test.py:415] workers restarted [worker-0]: I0421 03:49:39.965981 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24816", "localhost:18415", "localhost:18121", "localhost:24939"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I0421 03:49:39.990192 281472986018656 multi_process_runner.py:840] Subprocess with PID 1995541 (worker, 1) is now being started. [worker-1]: I0421 03:49:39.990637 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24816", "localhost:18415", "localhost:18121", "localhost:24939"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-3]: I0421 03:49:40.015381 281472986018656 multi_process_runner.py:840] Subprocess with PID 1995613 (worker, 3) is now being started. [worker-3]: I0421 03:49:40.015815 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24816", "localhost:18415", "localhost:18121", "localhost:24939"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-3]: 2023-04-21 03:49:40.056481: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:24939 [worker-0]: 2023-04-21 03:49:40.059217: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:24816 [worker-0]: 2023-04-21 03:49:40.070836: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 16454668516337298131 [worker-3]: 2023-04-21 03:49:40.071102: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: 2023-04-21 03:49:40.074092: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 1556304193204624130 [worker-0]: 2023-04-21 03:49:40.074532: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-2]: I0421 03:49:40.089111 281472986018656 multi_process_runner.py:840] Subprocess with PID 1995552 (worker, 2) is now being started. [worker-2]: I0421 03:49:40.089561 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:24816", "localhost:18415", "localhost:18121", "localhost:24939"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-1]: 2023-04-21 03:49:40.103138: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:18415 [worker-0]: 2023-04-21 03:49:40.118475: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 319244057445065593 [worker-1]: 2023-04-21 03:49:40.124027: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-2]: 2023-04-21 03:49:40.146409: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:18121 [worker-0]: 2023-04-21 03:49:40.157846: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 4412048651119776805 [worker-2]: 2023-04-21 03:49:40.158073: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0421 03:49:40.159802 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: I0421 03:49:40.159670 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-1]: I0421 03:49:40.159852 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0421 03:49:40.159837 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0421 03:49:40.231292 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0421 03:49:40.231754 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24816', 'localhost:18415', 'localhost:18121', 'localhost:24939']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0421 03:49:40.231929 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24816', 'localhost:18415', 'localhost:18121', 'localhost:24939']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0421 03:49:40.249971 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-2]: I0421 03:49:40.247576 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-3]: I0421 03:49:40.250388 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24816', 'localhost:18415', 'localhost:18121', 'localhost:24939']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0421 03:49:40.248025 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: I0421 03:49:40.250559 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24816', 'localhost:18415', 'localhost:18121', 'localhost:24939']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24816', 'localhost:18415', 'localhost:18121', 'localhost:24939']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0421 03:49:40.248202 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24816', 'localhost:18415', 'localhost:18121', 'localhost:24939']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0421 03:49:40.267823 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0421 03:49:40.268295 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24816', 'localhost:18415', 'localhost:18121', 'localhost:24939']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0421 03:49:40.268473 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:24816', 'localhost:18415', 'localhost:18121', 'localhost:24939']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0421 03:49:40.368295 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-0]: I0421 03:49:40.387429 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0421 03:49:40.393234 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: I0421 03:49:40.381782 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: I0421 03:49:40.405128 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-3]: I0421 03:49:40.412674 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-0]: Traceback (most recent call last): [worker-3]: Traceback (most recent call last): [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-0]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-3]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-1]: I0421 03:49:40.413805 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0421 03:49:40.413058 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: I0421 03:49:40.413362 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-3]: Instructions for updating: [worker-1]: Traceback (most recent call last): [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: self.run() [worker-3]: W0421 03:49:40.413817 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-3]: Instructions for updating: [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0421 03:49:40.413519 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: I0421 03:49:40.416688 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-0]: Instructions for updating: [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: Traceback (most recent call last): [worker-0]: INFO:tensorflow:Start training at 0 [worker-1]: self._target(*self._args, **self._kwargs) [worker-2]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-0]: I0421 03:49:40.413686 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: self.run() [worker-3]: self.run() [worker-1]: if self._termination_watcher_fn(): [worker-0]: self.run() [worker-3]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-0]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: self._target(*self._args, **self._kwargs) [worker-3]: INFO:tensorflow:Start training at 0 [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-3]: I0421 03:49:40.414216 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-0]: self._target(*self._args, **self._kwargs) [worker-2]: if self._termination_watcher_fn(): [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-3]: self._target(*self._args, **self._kwargs) [worker-1]: I0421 03:49:40.415733 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: if self._termination_watcher_fn(): [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: if self._termination_watcher_fn(): [worker-1]: Instructions for updating: [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: W0421 03:49:40.416020 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: Instructions for updating: [worker-2]: I0421 03:49:40.418661 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: INFO:tensorflow:Start training at 0 [worker-2]: Instructions for updating: [worker-1]: I0421 03:49:40.416207 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0421 03:49:40.418959 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0421 03:49:40.419120 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:40.534121 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:40.531418 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:40.533877 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:40.571929 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:40.675960 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:40.690233 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:40.698910 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:40.705048 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:40.834725 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:40.835015 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:40.834723 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:40.848545 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:40.959650 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:40.959331 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:40.971455 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:40.989953 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:41.120029 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:41.110325 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:41.139599 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:41.152655 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785e60d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0421 03:49:41.197350 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785e60d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:41.205568 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785ec0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0421 03:49:41.216847 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785ec0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785e60d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0421 03:49:41.219313 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785e60d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785ea0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0421 03:49:41.219451 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785ea0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:41.225047 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:41.237045 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:41.228611 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785e6940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0421 03:49:41.307751 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785e6940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:epoch 0 finished [worker-1]: I0421 03:49:41.308101 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785e6940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0421 03:49:41.306998 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785e6940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I0421 03:49:41.307352 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785ec940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0421 03:49:41.308022 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785ec940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:epoch 0 finished [worker-0]: I0421 03:49:41.308397 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785ea940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0421 03:49:41.312200 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785ea940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0421 03:49:41.312550 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:41.315443 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:41.316542 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:41.338406 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:41.350423 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:41.412778 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:41.430699 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:41.420450 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:41.420201 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:41.526875 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:41.533360 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:41.550244 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:41.557117 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:41.684006 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:41.693962 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:41.712113 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:41.690476 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:41.836329 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:41.840508 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:41.857328 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:41.851917 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:41.943457 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:41.947122 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:41.960362 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:41.980839 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0421 03:49:42.037037 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-2]: I0421 03:49:42.037308 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-1]: I0421 03:49:42.037249 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:42.045074 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I0421 03:49:42.036841 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:42.056993 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:42.060477 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:42.060286 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:42.162539 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:42.176613 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:42.192336 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:42.210701 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:42.285335 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:42.295132 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:42.310281 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:42.319208 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:42.422738 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:42.437739 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:42.458385 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:42.470262 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:42.522029 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:42.522119 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:42.526262 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:42.526933 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:42.576747 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:42.594925 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:42.598889 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:42.611398 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0421 03:49:42.657393 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-1]: I0421 03:49:42.657084 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: I0421 03:49:42.656876 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:42.665470 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:42.665504 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I0421 03:49:42.676616 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:42.677056 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:42.685083 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:42.746176 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:42.746557 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:42.742021 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:42.760421 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:42.834885 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:42.843338 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:42.855628 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:42.843497 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:42.909536 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:42.908714 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:42.925658 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:42.908326 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:42.992547 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:42.989153 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:43.014791 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:43.024707 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:43.105073 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:43.106183 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:43.103139 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:43.130490 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-3]: I0421 03:49:43.176409 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-1]: I0421 03:49:43.177456 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-0]: I0421 03:49:43.176789 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-2]: I0421 03:49:43.182102 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:43.186694 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:43.185841 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:43.200281 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:43.210114 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:43.352975 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:43.353380 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:43.366422 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:43.363011 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:43.442964 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:43.439941 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:43.547804 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:43.439527 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:43.640605 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:43.641228 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:43.644071 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:43.640636 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:43.735259 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:43.743641 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:43.748452 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:43.751391 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:43.836277 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:43.856828 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:43.860437 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:43.860408 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-3]: I0421 03:49:43.971651 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: INFO:tensorflow:Training finished. [worker-3]: I0421 03:49:43.971969 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-1]: INFO:tensorflow:epoch 4 finished [worker-2]: INFO:tensorflow:epoch 4 finished [worker-1]: I0421 03:49:43.973116 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: I0421 03:49:43.972902 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: INFO:tensorflow:Training finished. [worker-2]: INFO:tensorflow:Training finished. [worker-1]: I0421 03:49:43.973553 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-2]: I0421 03:49:43.973200 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-0]: I0421 03:49:43.971724 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I0421 03:49:43.972033 281472986018656 gce_failure_handler_test.py:244] Training finished. I0421 03:49:45.007079 281472889680736 multi_process_runner.py:646] worker-0 exit code: 0 I0421 03:49:45.007424 281472889680736 multi_process_runner.py:646] worker-1 exit code: 0 I0421 03:49:45.007566 281472889680736 multi_process_runner.py:646] worker-2 exit code: 0 I0421 03:49:45.007693 281472889680736 multi_process_runner.py:646] worker-3 exit code: 0 I0421 03:49:45.018945 281472889680736 multi_process_runner.py:662] Joining log reading threads. I0421 03:49:45.019349 281472889680736 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_0_inputarg_checkpoint_strategyoption_MWMSmultiworker): 11.5s I0421 03:49:45.176720 281472889680736 test_util.py:2462] time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_0_inputarg_checkpoint_strategyoption_MWMSmultiworker): 11.5s [ OK ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_0_inputarg_checkpoint_strategyoption_MWMSmultiworker [ RUN ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_7_inputarg_checkpoint_strategyoption_MWMSmultiworker INFO:tensorflow:Using local port 16944 I0421 03:49:45.178790 281472889680736 test_util.py:3794] Using local port 16944 INFO:tensorflow:Using local port 15659 I0421 03:49:45.179326 281472889680736 test_util.py:3794] Using local port 15659 INFO:tensorflow:Using local port 21518 I0421 03:49:45.179735 281472889680736 test_util.py:3794] Using local port 21518 INFO:tensorflow:Using local port 18843 I0421 03:49:45.180122 281472889680736 test_util.py:3794] Using local port 18843 INFO:tensorflow:Cluster starting. I0421 03:49:45.438237 281472889680736 gce_failure_handler_test.py:405] Cluster starting. [worker-0]: I0421 03:49:45.486512 281472986018656 multi_process_runner.py:840] Subprocess with PID 2001020 (worker, 0) is now being started. [worker-0]: I0421 03:49:45.486936 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:16944", "localhost:15659", "localhost:21518", "localhost:18843"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-1]: I0421 03:49:45.567673 281472986018656 multi_process_runner.py:840] Subprocess with PID 2001024 (worker, 1) is now being started. [worker-1]: I0421 03:49:45.568137 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:16944", "localhost:15659", "localhost:21518", "localhost:18843"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-2]: I0421 03:49:45.668502 281472986018656 multi_process_runner.py:840] Subprocess with PID 2001033 (worker, 2) is now being started. [worker-2]: I0421 03:49:45.669010 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:16944", "localhost:15659", "localhost:21518", "localhost:18843"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-3]: I0421 03:49:45.734898 281472986018656 multi_process_runner.py:840] Subprocess with PID 2001063 (worker, 3) is now being started. [worker-3]: I0421 03:49:45.735344 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:16944", "localhost:15659", "localhost:21518", "localhost:18843"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-0]: 2023-04-21 03:49:45.756840: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:16944 [worker-2]: 2023-04-21 03:49:45.759515: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:21518 [worker-1]: 2023-04-21 03:49:45.798715: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:15659 [worker-0]: 2023-04-21 03:49:45.808170: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 9387883937619202781 [worker-2]: 2023-04-21 03:49:45.808524: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: 2023-04-21 03:49:45.808382: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 1368932566112583878 [worker-0]: 2023-04-21 03:49:45.808611: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: 2023-04-21 03:49:45.810508: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 5400640774586588728 [worker-1]: 2023-04-21 03:49:45.811454: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-3]: 2023-04-21 03:49:46.277372: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:18843 [worker-0]: 2023-04-21 03:49:46.295528: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 5821103709894668948 [worker-3]: 2023-04-21 03:49:46.296418: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0421 03:49:46.332979 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0421 03:49:46.347628 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0421 03:49:46.367241 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0421 03:49:46.388247 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0421 03:49:46.453697 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0421 03:49:46.454304 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:16944', 'localhost:15659', 'localhost:21518', 'localhost:18843']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0421 03:49:46.454487 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:16944', 'localhost:15659', 'localhost:21518', 'localhost:18843']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0421 03:49:46.499708 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0421 03:49:46.500217 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:16944', 'localhost:15659', 'localhost:21518', 'localhost:18843']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0421 03:49:46.500395 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:16944', 'localhost:15659', 'localhost:21518', 'localhost:18843']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0421 03:49:46.532309 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-1]: I0421 03:49:46.532916 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:16944', 'localhost:15659', 'localhost:21518', 'localhost:18843']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0421 03:49:46.533105 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:16944', 'localhost:15659', 'localhost:21518', 'localhost:18843']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0421 03:49:46.543298 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-0]: I0421 03:49:46.543775 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:16944', 'localhost:15659', 'localhost:21518', 'localhost:18843']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0421 03:49:46.543950 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:16944', 'localhost:15659', 'localhost:21518', 'localhost:18843']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0421 03:49:46.703192 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0421 03:49:46.703212 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-3]: I0421 03:49:46.704485 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0421 03:49:46.715654 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0421 03:49:46.719301 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-3]: Traceback (most recent call last): [worker-3]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0421 03:49:46.705018 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-3]: Instructions for updating: [worker-0]: I0421 03:49:46.727478 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0421 03:49:46.705420 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0421 03:49:46.705799 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: self.run() [worker-3]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-3]: if self._termination_watcher_fn(): [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0421 03:49:46.744840 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-2]: Traceback (most recent call last): [worker-2]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-2]: self.run() [worker-2]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-2]: self._target(*self._args, **self._kwargs) [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: if self._termination_watcher_fn(): [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0421 03:49:46.734083 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0421 03:49:46.734554 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-0]: Traceback (most recent call last): [worker-0]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-0]: self.run() [worker-0]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-0]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: if self._termination_watcher_fn(): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0421 03:49:46.748532 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0421 03:49:46.749039 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0421 03:49:46.749223 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0421 03:49:46.734721 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-1]: I0421 03:49:46.846902 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-1]: Traceback (most recent call last): [worker-1]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0421 03:49:46.847670 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0421 03:49:46.848035 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0421 03:49:46.848229 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: self.run() [worker-1]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: if self._termination_watcher_fn(): [worker-3]: I0421 03:49:46.849947 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:46.905722 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:46.971064 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:47.006519 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:47.132352 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:47.107964 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:47.121127 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:47.147068 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:47.247161 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:47.253420 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:47.250171 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:47.277641 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:47.440345 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:47.440856 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:47.451445 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:47.450246 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:47.548847 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:47.600906 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:47.680282 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:47.700683 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785eb0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785eb0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785ed0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785ec0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0421 03:49:47.824489 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785eb0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0421 03:49:47.824710 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785eb0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0421 03:49:47.824962 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785ec0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0421 03:49:47.824871 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785ed0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:47.832996 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:47.833831 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:47.833890 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:47.834433 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785eb940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0421 03:49:47.907170 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785eb940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I0421 03:49:47.907555 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:47.915668 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785eb940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785ed940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0421 03:49:47.926527 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785ed940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:epoch 0 finished [worker-1]: I0421 03:49:47.926922 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785ec940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0421 03:49:47.923138 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785ec940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0421 03:49:47.917497 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785eb940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: INFO:tensorflow:epoch 0 finished [worker-2]: INFO:tensorflow:epoch 0 finished [worker-0]: I0421 03:49:47.917905 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: I0421 03:49:47.923543 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:47.935349 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:47.950773 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:47.972445 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:48.107106 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:48.134292 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:48.137168 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:48.155426 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:48.216155 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:48.216927 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:48.232665 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:48.226370 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:48.317080 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:48.328749 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:48.324467 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:48.364066 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:48.447770 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:48.461365 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:48.473516 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:48.488162 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:48.590575 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:48.594459 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:48.610271 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:48.631008 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-0]: I0421 03:49:48.677586 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: I0421 03:49:48.677920 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: INFO:tensorflow:epoch 1 finished [worker-3]: I0421 03:49:48.677323 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-1]: I0421 03:49:48.677974 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:48.686526 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:48.686996 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:48.687701 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:48.687785 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:48.824932 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:48.842598 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:48.842521 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:48.853677 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:48.960442 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:48.970059 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:48.970834 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:49.058194 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:49.163208 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:49.168411 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:49.195438 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:49.191543 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:49.310977 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:49.311209 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:49.318103 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:49.311279 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:49.462012 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:49.478556 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:49.477856 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:49.514274 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 2 finished [worker-0]: INFO:tensorflow:epoch 2 finished [worker-3]: I0421 03:49:49.575259 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: I0421 03:49:49.575574 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-2]: INFO:tensorflow:epoch 2 finished [worker-2]: I0421 03:49:49.575846 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:49.584799 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:49.575700 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: I0421 03:49:49.586031 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:49.583818 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:49.583625 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:49.641026 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:49.642361 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:49.643449 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:49.641693 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:49.695063 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:49.694018 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:49.717147 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:49.723099 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:49.782122 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:49.778801 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:49.797547 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:49.789311 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:49.874297 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:49.884305 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:49.891014 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:49.896108 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:49.964990 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:49.990451 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:50.000666 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:49.991295 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-3]: I0421 03:49:50.071761 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-0]: I0421 03:49:50.072005 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-2]: I0421 03:49:50.073702 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-1]: I0421 03:49:50.072155 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:50.080606 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:50.081288 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:50.080398 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:50.102953 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:50.160504 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:50.158042 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:50.160476 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:50.164619 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:50.217217 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:50.322684 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:50.218619 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:50.218151 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:50.404672 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:50.406820 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:50.413282 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:50.450415 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:50.533249 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:50.533629 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:50.551959 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:50.561625 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:50.633821 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:50.639399 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:50.650531 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:50.651364 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 4 finished [worker-0]: I0421 03:49:50.710157 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: INFO:tensorflow:Training finished. [worker-0]: I0421 03:49:50.710565 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-1]: INFO:tensorflow:epoch 4 finished [worker-3]: INFO:tensorflow:epoch 4 finished [worker-1]: I0421 03:49:50.712941 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: I0421 03:49:50.712738 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-1]: INFO:tensorflow:Training finished. [worker-2]: INFO:tensorflow:epoch 4 finished [worker-3]: INFO:tensorflow:Training finished. [worker-1]: I0421 03:49:50.713236 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-2]: I0421 03:49:50.713794 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: I0421 03:49:50.713039 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-2]: INFO:tensorflow:Training finished. [worker-2]: I0421 03:49:50.714148 281472986018656 gce_failure_handler_test.py:244] Training finished. INFO:tensorflow:restarting workers I0421 03:49:52.530815 281472889680736 gce_failure_handler_test.py:411] restarting workers INFO:tensorflow:workers restarted I0421 03:49:52.636544 281472889680736 gce_failure_handler_test.py:415] workers restarted [worker-0]: I0421 03:49:52.742000 281472986018656 multi_process_runner.py:840] Subprocess with PID 2009446 (worker, 0) is now being started. [worker-2]: I0421 03:49:52.742569 281472986018656 multi_process_runner.py:840] Subprocess with PID 2009452 (worker, 2) is now being started. [worker-3]: I0421 03:49:52.744231 281472986018656 multi_process_runner.py:840] Subprocess with PID 2009456 (worker, 3) is now being started. [worker-2]: I0421 03:49:52.742927 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:16944", "localhost:15659", "localhost:21518", "localhost:18843"]}, "task": {"type": "worker", "index": 2}, "rpc_layer": "grpc"}' [worker-0]: I0421 03:49:52.742440 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:16944", "localhost:15659", "localhost:21518", "localhost:18843"]}, "task": {"type": "worker", "index": 0}, "rpc_layer": "grpc"}' [worker-3]: I0421 03:49:52.744748 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:16944", "localhost:15659", "localhost:21518", "localhost:18843"]}, "task": {"type": "worker", "index": 3}, "rpc_layer": "grpc"}' [worker-1]: I0421 03:49:52.749157 281472986018656 multi_process_runner.py:840] Subprocess with PID 2009450 (worker, 1) is now being started. [worker-1]: I0421 03:49:52.749633 281472986018656 multi_process_runner.py:842] TF_CONFIG: '{"cluster": {"worker": ["localhost:16944", "localhost:15659", "localhost:21518", "localhost:18843"]}, "task": {"type": "worker", "index": 1}, "rpc_layer": "grpc"}' [worker-0]: 2023-04-21 03:49:52.776845: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:16944 [worker-2]: 2023-04-21 03:49:52.777011: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:21518 [worker-0]: 2023-04-21 03:49:52.792958: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 5124401364976057729 [worker-0]: 2023-04-21 03:49:52.793027: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:2 has connected to coordination service. Incarnation: 8429547520202530797 [worker-0]: 2023-04-21 03:49:52.793432: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-2]: 2023-04-21 03:49:52.794160: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-1]: 2023-04-21 03:49:52.822208: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:15659 [worker-0]: 2023-04-21 03:49:52.826094: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 6771659720378493116 [worker-1]: 2023-04-21 03:49:52.826349: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-3]: 2023-04-21 03:49:52.861173: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:449] Started server with target: grpc://localhost:18843 [worker-0]: 2023-04-21 03:49:52.910647: I tensorflow/tsl/distributed_runtime/coordination/coordination_service.cc:535] /job:worker/replica:0/task:3 has connected to coordination service. Incarnation: 3047019751580334080 [worker-3]: 2023-04-21 03:49:52.911733: I tensorflow/tsl/distributed_runtime/coordination/coordination_service_agent.cc:298] Coordination agent has successfully connected. [worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-0]: I0421 03:49:52.915612 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-1]: I0421 03:49:52.918655 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-3]: I0421 03:49:52.937446 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1'] [worker-2]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-2]: I0421 03:49:52.927369 281472986018656 collective_all_reduce_strategy.py:531] Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:1', '/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:CPU:1', '/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:1', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:1'] [worker-3]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: I0421 03:49:53.002074 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:3/device:CPU:0',) [worker-3]: INFO:tensorflow:Check health not enabled. [worker-3]: I0421 03:49:53.002673 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-3]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:16944', 'localhost:15659', 'localhost:21518', 'localhost:18843']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-3]: I0421 03:49:53.002854 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:16944', 'localhost:15659', 'localhost:21518', 'localhost:18843']}, task_type = 'worker', task_id = 3, num_workers = 4, local_devices = ('/job:worker/task:3/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: I0421 03:49:53.041370 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:2/device:CPU:0',) [worker-2]: INFO:tensorflow:Check health not enabled. [worker-2]: I0421 03:49:53.041818 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-2]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:16944', 'localhost:15659', 'localhost:21518', 'localhost:18843']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: I0421 03:49:53.041990 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:16944', 'localhost:15659', 'localhost:21518', 'localhost:18843']}, task_type = 'worker', task_id = 2, num_workers = 4, local_devices = ('/job:worker/task:2/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: I0421 03:49:53.051309 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:0/device:CPU:0',) [worker-0]: INFO:tensorflow:Check health not enabled. [worker-1]: INFO:tensorflow:Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: I0421 03:49:53.040190 281472986018656 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:worker/task:1/device:CPU:0',) [worker-1]: INFO:tensorflow:Check health not enabled. [worker-0]: I0421 03:49:53.051770 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-1]: I0421 03:49:53.040765 281472986018656 collective_all_reduce_strategy.py:574] Check health not enabled. [worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:16944', 'localhost:15659', 'localhost:21518', 'localhost:18843']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:16944', 'localhost:15659', 'localhost:21518', 'localhost:18843']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-1]: I0421 03:49:53.040942 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:16944', 'localhost:15659', 'localhost:21518', 'localhost:18843']}, task_type = 'worker', task_id = 1, num_workers = 4, local_devices = ('/job:worker/task:1/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-0]: I0421 03:49:53.051943 281472986018656 collective_all_reduce_strategy.py:576] MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:16944', 'localhost:15659', 'localhost:21518', 'localhost:18843']}, task_type = 'worker', task_id = 0, num_workers = 4, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.AUTO [worker-2]: INFO:tensorflow:Start watcher for peer's signal. [worker-2]: I0421 03:49:53.155587 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-0]: INFO:tensorflow:Start watcher for peer's signal. [worker-0]: I0421 03:49:53.162062 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start watcher for peer's signal. [worker-1]: I0421 03:49:53.169437 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-1]: INFO:tensorflow:Start polling for termination signal. [worker-1]: I0421 03:49:53.170324 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-1]: Exception in thread WorkerTerminationSignalWatcher-1: [worker-1]: Traceback (most recent call last): [worker-1]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-1]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-1]: I0421 03:49:53.170808 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-1]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: W0421 03:49:53.171121 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-1]: Instructions for updating: [worker-1]: Track steps using a tf.Variable saved in checkpoint instead. [worker-1]: INFO:tensorflow:Start training at 0 [worker-1]: I0421 03:49:53.171285 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: self.run() [worker-1]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-1]: self._target(*self._args, **self._kwargs) [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-1]: if self._termination_watcher_fn(): [worker-1]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-1]: elif frequent_send and not maintenance_event.is_set(): [worker-1]: AttributeError: 'str' object has no attribute 'is_set' [worker-0]: INFO:tensorflow:Start polling for termination signal. [worker-0]: I0421 03:49:53.176668 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-3]: INFO:tensorflow:Start watcher for peer's signal. [worker-3]: I0421 03:49:53.180131 281472986018656 failure_handling.py:634] Start watcher for peer's signal. [worker-2]: INFO:tensorflow:Start polling for termination signal. [worker-2]: I0421 03:49:53.176664 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-3]: INFO:tensorflow:Start polling for termination signal. [worker-0]: Exception in thread WorkerTerminationSignalWatcher-0: [worker-3]: I0421 03:49:53.197085 281472986018656 failure_handling.py:683] Start polling for termination signal. [worker-0]: Traceback (most recent call last): [worker-0]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-0]: self.run() [worker-0]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-2]: Exception in thread WorkerTerminationSignalWatcher-2: [worker-0]: self._target(*self._args, **self._kwargs) [worker-2]: Traceback (most recent call last): [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-2]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-2]: self.run() [worker-2]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-0]: if self._termination_watcher_fn(): [worker-2]: self._target(*self._args, **self._kwargs) [worker-0]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-0]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: if self._termination_watcher_fn(): [worker-0]: AttributeError: 'str' object has no attribute 'is_set' [worker-2]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-2]: elif frequent_send and not maintenance_event.is_set(): [worker-2]: AttributeError: 'str' object has no attribute 'is_set' [worker-0]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-0]: I0421 03:49:53.203195 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-0]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: W0421 03:49:53.203613 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-0]: Instructions for updating: [worker-0]: Track steps using a tf.Variable saved in checkpoint instead. [worker-0]: INFO:tensorflow:Start training at 0 [worker-0]: I0421 03:49:53.203785 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-2]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-2]: I0421 03:49:53.216203 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-2]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: W0421 03:49:53.216632 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-2]: Instructions for updating: [worker-2]: Track steps using a tf.Variable saved in checkpoint instead. [worker-2]: INFO:tensorflow:Start training at 0 [worker-2]: I0421 03:49:53.216802 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-3]: Exception in thread WorkerTerminationSignalWatcher-3: [worker-3]: Traceback (most recent call last): [worker-3]: File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner [worker-3]: self.run() [worker-3]: File "/usr/lib/python3.8/threading.py", line 870, in run [worker-3]: self._target(*self._args, **self._kwargs) [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/failure_handling.py", line 692, in _poll_termination_signal [worker-3]: if self._termination_watcher_fn(): [worker-3]: File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 145, in mock_termination_watcher_function_gce [worker-3]: elif frequent_send and not maintenance_event.is_set(): [worker-3]: AttributeError: 'str' object has no attribute 'is_set' [worker-3]: INFO:tensorflow:PreemptionCheckpointHandler initialized or restored. [worker-3]: I0421 03:49:53.217942 281472986018656 failure_handling.py:538] PreemptionCheckpointHandler initialized or restored. [worker-3]: WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: W0421 03:49:53.218444 281472986018656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py:195: PreemptionCheckpointHandler.total_run_calls (from tensorflow.python.distribute.failure_handling.failure_handling) is deprecated and will be removed in a future version. [worker-3]: Instructions for updating: [worker-3]: Track steps using a tf.Variable saved in checkpoint instead. [worker-3]: INFO:tensorflow:Start training at 0 [worker-3]: I0421 03:49:53.218614 281472986018656 gce_failure_handler_test.py:194] Start training at 0 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:53.326274 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:53.346387 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:53.340178 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:53.340402 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:53.443892 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:53.445389 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:53.445768 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:53.446225 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:53.509851 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:53.510702 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:53.519155 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:53.519894 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:53.597922 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:53.608339 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:53.609017 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:53.608677 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:53.662938 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:53.667819 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:53.677022 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:53.669440 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785e50d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785e40d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785ea0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0421 03:49:53.724946 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785e50d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0421 03:49:53.725285 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785e40d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0421 03:49:53.725108 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785ea0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: WARNING:tensorflow:5 out of the last 5 calls to .wrapped_fn at 0xffff785e90d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0421 03:49:53.730029 281472986018656 polymorphic_function.py:158] 5 out of the last 5 calls to .wrapped_fn at 0xffff785e90d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:53.733820 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:53.740332 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:53.735308 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:53.734050 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785e4940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785ea940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: W0421 03:49:53.789930 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785e4940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-0]: W0421 03:49:53.790158 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785ea940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-1]: INFO:tensorflow:epoch 0 finished [worker-0]: INFO:tensorflow:epoch 0 finished [worker-1]: I0421 03:49:53.790335 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: I0421 03:49:53.790518 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-3]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785e5940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: W0421 03:49:53.789281 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785e5940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-3]: INFO:tensorflow:epoch 0 finished [worker-3]: I0421 03:49:53.789690 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-2]: WARNING:tensorflow:6 out of the last 6 calls to .wrapped_fn at 0xffff785e9940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: W0421 03:49:53.795833 281472986018656 polymorphic_function.py:158] 6 out of the last 6 calls to .wrapped_fn at 0xffff785e9940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [worker-2]: INFO:tensorflow:epoch 0 finished [worker-2]: I0421 03:49:53.796366 281472986018656 gce_failure_handler_test.py:192] epoch 0 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:53.799193 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:53.799428 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:53.799551 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:53.812431 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:53.868890 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:53.875866 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:53.876783 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:53.881598 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:53.933383 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:53.936974 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:53.950746 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:53.954564 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:54.142032 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:54.136811 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:54.149437 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:54.143091 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:54.199976 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:54.200143 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:54.201918 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:54.220433 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:54.324564 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:54.335788 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:54.337184 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:54.360166 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 1 finished [worker-3]: I0421 03:49:54.421614 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:epoch 1 finished [worker-0]: I0421 03:49:54.421722 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:54.430134 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:epoch 1 finished [worker-1]: I0421 03:49:54.436910 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-2]: INFO:tensorflow:epoch 1 finished [worker-2]: I0421 03:49:54.436724 281472986018656 gce_failure_handler_test.py:192] epoch 1 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:54.442461 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:54.450668 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:54.482986 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:54.556840 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:54.566338 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:54.569712 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:54.581627 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:54.677957 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:54.679090 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:54.690639 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:54.710904 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:54.785971 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:54.784466 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:54.852839 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:54.859915 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:54.930999 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:54.936288 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:54.935785 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:54.950439 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:55.032956 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:55.033786 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:55.050427 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:55.080034 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:epoch 2 finished [worker-0]: I0421 03:49:55.157071 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-1]: INFO:tensorflow:epoch 2 finished [worker-3]: INFO:tensorflow:epoch 2 finished [worker-3]: I0421 03:49:55.156959 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:epoch 2 finished [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:55.162801 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-0]: I0421 03:49:55.167135 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:55.172069 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:55.172719 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:55.176794 281472986018656 gce_failure_handler_test.py:192] epoch 2 finished [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:55.210461 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:55.320784 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:55.305040 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:55.320199 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:55.347599 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:55.421563 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:55.430536 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:55.460902 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:55.461135 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:55.543639 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:55.549521 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:55.560686 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:55.570337 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:55.623355 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:55.623642 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:55.640224 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:55.640833 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:55.714441 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:55.722225 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:55.713923 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:55.719270 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 3 finished [worker-0]: INFO:tensorflow:epoch 3 finished [worker-3]: I0421 03:49:55.777493 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: I0421 03:49:55.778047 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-2]: INFO:tensorflow:epoch 3 finished [worker-2]: I0421 03:49:55.783231 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-1]: INFO:tensorflow:epoch 3 finished [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:55.797232 281472986018656 gce_failure_handler_test.py:192] epoch 3 finished [worker-0]: I0421 03:49:55.787404 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:55.787050 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:55.819954 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:55.830053 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:55.928229 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:55.932519 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:55.936640 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:55.940569 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:56.106495 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:56.110488 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:56.016653 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:56.133735 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:56.195291 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:56.195574 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:56.196181 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:56.215931 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:56.274696 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:56.274827 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:56.275737 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:56.297044 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-0]: I0421 03:49:56.374861 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-2]: I0421 03:49:56.374957 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: I0421 03:49:56.375672 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-1]: I0421 03:49:56.377206 281472986018656 cross_device_ops.py:1151] Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 4, implementation = CommunicationImplementation.AUTO, num_packs = 1 [worker-3]: INFO:tensorflow:epoch 4 finished [worker-3]: I0421 03:49:56.417227 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: INFO:tensorflow:epoch 4 finished [worker-3]: INFO:tensorflow:Training finished. [worker-0]: I0421 03:49:56.418139 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-3]: I0421 03:49:56.417536 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-0]: INFO:tensorflow:Training finished. [worker-2]: INFO:tensorflow:epoch 4 finished [worker-2]: I0421 03:49:56.418905 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-0]: I0421 03:49:56.418427 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-1]: INFO:tensorflow:epoch 4 finished [worker-2]: INFO:tensorflow:Training finished. [worker-1]: I0421 03:49:56.419433 281472986018656 gce_failure_handler_test.py:192] epoch 4 finished [worker-2]: I0421 03:49:56.419167 281472986018656 gce_failure_handler_test.py:244] Training finished. [worker-1]: INFO:tensorflow:Training finished. [worker-1]: I0421 03:49:56.419749 281472986018656 gce_failure_handler_test.py:244] Training finished. I0421 03:49:56.607098 281472889680736 multi_process_runner.py:646] worker-0 exit code: 0 I0421 03:49:56.607417 281472889680736 multi_process_runner.py:646] worker-1 exit code: 0 I0421 03:49:56.607555 281472889680736 multi_process_runner.py:646] worker-2 exit code: 0 I0421 03:49:56.607677 281472889680736 multi_process_runner.py:646] worker-3 exit code: 0 I0421 03:49:56.618614 281472889680736 multi_process_runner.py:662] Joining log reading threads. I0421 03:49:56.618956 281472889680736 multi_process_runner.py:665] Joined log reading threads. INFO:tensorflow:time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_7_inputarg_checkpoint_strategyoption_MWMSmultiworker): 11.59s I0421 03:49:56.768692 281472889680736 test_util.py:2462] time(__main__.GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_7_inputarg_checkpoint_strategyoption_MWMSmultiworker): 11.59s [ OK ] GceFailureHandlingTest.test_multiple_workers_preempted_consecutively_test_apiwrappingtrain_True_graceperiod_7_inputarg_checkpoint_strategyoption_MWMSmultiworker ====================================================================== FAIL: test_basic_run_test_inputarg_checkpoint_strategyoption_MWMSmultiworker (__main__.GceFailureHandlingTest) GceFailureHandlingTest.test_basic_run_test_inputarg_checkpoint_strategyoption_MWMSmultiworker test_basic_run_test_inputarg_checkpoint_strategyoption_MWMSmultiworker(input_arg='checkpoint', strategy_option='MWMS_multi_worker') ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/parameterized.py", line 314, in bound_param_test return test_method(self, **testcase_params) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 360, in decorated execute_test_method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 343, in execute_test_method test_method(**kwargs_to_pass) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 559, in decorator test_method(self, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 329, in test_basic_run mpr.join(timeout=250) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 649, in join self._reraise_if_subprocess_error(process_statuses) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 565, in _reraise_if_subprocess_error six.reraise(*process_status.exc_info) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/six_archive/six.py", line 719, in reraise raise value File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/multi_process_runner.py", line 1060, in _run_contained return_value = fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/org_tensorflow/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.py", line 211, in worker_fn self.assertNotEmpty(checkpoint_index) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 972, in assertNotEmpty self.fail('{!r} has length of 0.'.format(container), msg) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/distribute/failure_handling/gce_failure_handler_test.runfiles/absl_py/absl/testing/absltest.py", line 1814, in fail return super(TestCase, self).fail(self._formatMessage(prefix, msg)) File "/usr/lib/python3.8/unittest/case.py", line 753, in fail raise self.failureException(msg) AssertionError: [] has length of 0. ---------------------------------------------------------------------- Ran 7 tests in 84.695s FAILED (failures=1) ================================================================================ ==================== Test output for //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu (shard 9 of 20): 2023-04-21 03:58:07.302876: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:266: ComplexWarning: Casting complex values to real discards the imaginary part a_np.imag = np.random.normal(-5, 5, /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:270: ComplexWarning: Casting complex values to real discards the imaginary part b_np.imag = np.random.normal(-5, 5, Running tests under Python 3.8.10: /usr/local/bin/python3 [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_True_False WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:153: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.config.list_physical_devices('GPU')` instead. W0421 03:58:12.389419 281473367569248 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:153: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.config.list_physical_devices('GPU')` instead. INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_True_False): 0.01s I0421 03:58:12.395345 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_True_False): 0.01s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_True): 0.0s I0421 03:58:12.396250 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_True_False_False): 0.0s I0421 03:58:12.396909 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_True_False): 0.0s I0421 03:58:12.397508 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_True): 0.0s I0421 03:58:12.398105 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_1_False_False_False_False): 0.0s I0421 03:58:12.398692 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_1_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_True_False): 0.0s I0421 03:58:12.399277 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_True): 0.0s I0421 03:58:12.399857 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_True_False_False_False): 0.0s I0421 03:58:12.400440 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_True_False): 0.0s I0421 03:58:12.401024 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_True): 0.0s I0421 03:58:12.401607 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_True_False_False): 0.0s I0421 03:58:12.402193 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_True_False 2023-04-21 03:58:12.441857: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:375] MLIR V1 optimization pass is not enabled INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_True_False): 0.16s I0421 03:58:12.558104 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_True_False): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_True): 0.12s I0421 03:58:12.683144 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_True): 0.12s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_5_False_False_False_False): 0.19s I0421 03:58:12.870709 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_5_False_False_False_False): 0.19s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_True_False): 0.18s I0421 03:58:13.048571 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_True_False): 0.18s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_True): 0.1s I0421 03:58:13.145525 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_True): 0.1s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_True_False_False_False): 0.17s I0421 03:58:13.317280 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_True_False_False_False): 0.17s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_True_False): 0.19s I0421 03:58:13.509808 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_True_False): 0.19s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_True): 0.22s I0421 03:58:13.734471 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_True): 0.22s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_True_False_False): 0.11s I0421 03:58:13.844511 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_True_False_False): 0.11s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_True_False): 0.22s I0421 03:58:14.061314 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_True_False): 0.22s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_True): 0.25s I0421 03:58:14.308149 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_True): 0.25s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_3_False_False_False_False): 0.34s I0421 03:58:14.646716 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_3_False_False_False_False): 0.34s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_3_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_True_False): 0.19s I0421 03:58:14.835741 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_True_False): 0.19s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_True): 0.13s I0421 03:58:14.968532 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_True): 0.13s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_True_False_False_False): 0.14s I0421 03:58:15.111253 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_True_False_False_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_True_False): 0.15s I0421 03:58:15.257466 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_True_False): 0.15s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_True): 0.1s I0421 03:58:15.355928 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_True): 0.1s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_True_False_False): 0.16s I0421 03:58:15.517069 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_True_False_False): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_True_False): 0.23s I0421 03:58:15.747473 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_True_False): 0.23s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_True): 0.36s I0421 03:58:16.108535 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_True): 0.36s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_1_False_False_False_False): 0.12s I0421 03:58:16.227328 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_1_False_False_False_False): 0.12s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_True_False): 0.24s I0421 03:58:16.468555 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_True_False): 0.24s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_True): 0.22s I0421 03:58:16.693044 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_True): 0.22s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_True_False_False_False): 0.39s I0421 03:58:17.086726 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_True_False_False_False): 0.39s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_True_False): 0.42s I0421 03:58:17.509060 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_True_False): 0.42s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_True): 0.0s I0421 03:58:17.509976 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_True_False_False): 0.0s I0421 03:58:17.510564 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_True_False): 0.0s I0421 03:58:17.511111 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_True): 0.0s I0421 03:58:17.511641 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_5_False_False_False_False): 0.0s I0421 03:58:17.512163 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_5_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_True_False): 0.0s I0421 03:58:17.512683 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_True): 0.0s I0421 03:58:17.513205 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_True_False_False_False): 0.0s I0421 03:58:17.513723 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_True_False): 0.0s I0421 03:58:17.514237 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_True): 0.0s I0421 03:58:17.514762 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_True_False_False): 0.0s I0421 03:58:17.515275 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_True_False): 0.0s I0421 03:58:17.515786 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_True): 0.15s I0421 03:58:17.670683 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_True): 0.15s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_3_False_False_False_False): 0.2s I0421 03:58:17.874041 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_3_False_False_False_False): 0.2s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_3_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_True_False): 0.14s I0421 03:58:18.014455 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_True_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_True): 0.18s I0421 03:58:18.194750 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_True): 0.18s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_True_False_False_False): 0.14s I0421 03:58:18.334478 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_True_False_False_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_True_False): 0.14s I0421 03:58:18.473723 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_True_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_True): 0.15s I0421 03:58:18.625152 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_True): 0.15s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_True_False_False): 0.29s I0421 03:58:18.916363 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_True_False_False): 0.29s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_True_False): 0.29s I0421 03:58:19.207699 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_True_False): 0.29s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_True): 0.18s I0421 03:58:19.389186 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_True): 0.18s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_1_False_False_False_False): 0.27s I0421 03:58:19.658881 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_1_False_False_False_False): 0.27s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_True_False): 0.3s I0421 03:58:19.957012 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_True_False): 0.3s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_True): 0.08s I0421 03:58:20.041036 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_True): 0.08s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_True_False_False_False): 0.08s I0421 03:58:20.126209 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_True_False_False_False): 0.08s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_True_False): 0.1s I0421 03:58:20.226103 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_True_False): 0.1s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_True): 0.13s I0421 03:58:20.352062 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_True): 0.13s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_True_False_False): 0.1s I0421 03:58:20.454506 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_True_False_False): 0.1s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_True_False): 0.11s I0421 03:58:20.562234 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_True_False): 0.11s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_True): 0.16s I0421 03:58:20.721797 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_True): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_5_False_False_False_False): 0.17s I0421 03:58:20.889326 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_5_False_False_False_False): 0.17s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_True_False): 0.11s I0421 03:58:20.997147 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_True_False): 0.11s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_True): 0.12s I0421 03:58:21.114701 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_True): 0.12s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_True_False_False_False): 0.15s I0421 03:58:21.268912 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_True_False_False_False): 0.15s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_True_False): 0.17s I0421 03:58:21.439382 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_True_False): 0.17s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_True): 0.0s I0421 03:58:21.440216 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_True_False_False): 0.0s I0421 03:58:21.440781 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_True_False): 0.0s I0421 03:58:21.441314 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_True): 0.0s I0421 03:58:21.441836 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_3_False_False_False_False): 0.0s I0421 03:58:21.442355 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_3_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_3_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_True_False): 0.0s I0421 03:58:21.442869 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_True): 0.0s I0421 03:58:21.443384 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_True_False_False_False): 0.0s I0421 03:58:21.443907 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_True_False): 0.0s I0421 03:58:21.444422 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_True): 0.0s I0421 03:58:21.444930 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_True_False_False): 0.0s I0421 03:58:21.445441 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_True_False): 0.0s I0421 03:58:21.445949 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_True): 0.0s I0421 03:58:21.446509 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_1_False_False_False_False): 0.0s I0421 03:58:21.447022 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_1_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_True_False): 0.0s I0421 03:58:21.447535 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_True): 0.0s I0421 03:58:21.448043 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_True_False_False_False): 0.0s I0421 03:58:21.448565 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_True_False): 0.0s I0421 03:58:21.449073 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_True): 0.0s I0421 03:58:21.449586 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_True_False_False): 0.0s I0421 03:58:21.450093 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_True_False): 0.0s I0421 03:58:21.450602 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_True): 0.0s I0421 03:58:21.451109 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_5_False_False_False_False): 0.0s I0421 03:58:21.451617 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_5_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_True_False): 0.0s I0421 03:58:21.452124 281473367569248 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_True_False [ RUN ] MatMulInfixOperatorTest.testMismatchedShape INFO:tensorflow:time(__main__.MatMulInfixOperatorTest.testMismatchedShape): 0.04s I0421 03:58:21.488109 281473367569248 test_util.py:2462] time(__main__.MatMulInfixOperatorTest.testMismatchedShape): 0.04s [ FAILED ] MatMulInfixOperatorTest.testMismatchedShape [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_True_False_False_False 2023-04-21 03:58:21.499954: E ./tensorflow/core/graph/mkl_graph_util.h:182] oneDNN BFloat16 support are only on platforms with AVX512. Falling back to default implementation if present. /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:103: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. np_val = np.matrix(a_np_) * np.matrix(b_np_) /usr/local/lib/python3.8/dist-packages/numpy/matrixlib/defmatrix.py:69: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. return matrix(data, dtype=dtype, copy=False) INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_True_False_False_False): 0.01s I0421 03:58:21.503131 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_True_False): 0.01s I0421 03:58:21.511106 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_True): 0.01s I0421 03:58:21.518202 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_True_False_False): 0.01s I0421 03:58:21.525553 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_True_False): 0.01s I0421 03:58:21.532901 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_True): 0.01s I0421 03:58:21.540013 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_5_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_5_1_False_False_False_False): 0.02s I0421 03:58:21.556451 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_5_1_False_False_False_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_5_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_True_False): 0.01s I0421 03:58:21.563659 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_True): 0.01s I0421 03:58:21.570556 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_True_False_False_False): 0.01s I0421 03:58:21.578008 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_True_False): 0.01s I0421 03:58:21.585010 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_True): 0.01s I0421 03:58:21.592893 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_True_False_False): 0.01s I0421 03:58:21.600054 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_True_False): 0.01s I0421 03:58:21.613690 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_True): 0.01s I0421 03:58:21.627020 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_5_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_5_5_False_False_False_False): 0.01s I0421 03:58:21.634182 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_5_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_5_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_True_False): 0.01s I0421 03:58:21.647099 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_True): 0.01s I0421 03:58:21.654255 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_3_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_5_True_False_False_False): 0.01s I0421 03:58:21.668006 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_5_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_3_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_True_False): 0.01s I0421 03:58:21.682076 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_True): 0.01s I0421 03:58:21.689384 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_True_False_False): 0.01s I0421 03:58:21.696614 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_True_False): 0.01s I0421 03:58:21.709408 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_True): 0.01s I0421 03:58:21.722251 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_1_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_1_3_False_False_False_False): 0.01s I0421 03:58:21.729371 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_1_3_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_1_3_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_True_False): 0.01s I0421 03:58:21.742223 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_True): 0.01s I0421 03:58:21.753407 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_5_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_5_3_True_False_False_False): 0.01s I0421 03:58:21.766729 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_5_3_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_5_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_True_False): 0.01s I0421 03:58:21.780897 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_True): 0.01s I0421 03:58:21.788094 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_True_False_False): 0.01s I0421 03:58:21.795303 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_True_False): 0.01s I0421 03:58:21.808979 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_True): 0.01s I0421 03:58:21.822159 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_3_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_3_1_False_False_False_False): 0.01s I0421 03:58:21.829444 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_3_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_3_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_True_False): 0.01s I0421 03:58:21.843102 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_True): 0.01s I0421 03:58:21.850501 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_1_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_1_True_False_False_False): 0.01s I0421 03:58:21.860299 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_1_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_True_False): 0.01s I0421 03:58:21.867354 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_True): 0.01s I0421 03:58:21.874362 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_5_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_5_3_False_True_False_False): 0.01s I0421 03:58:21.881438 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_5_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_5_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_True_False): 0.01s I0421 03:58:21.888685 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_True): 0.01s I0421 03:58:21.895601 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_3_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_3_5_False_False_False_False): 0.01s I0421 03:58:21.902576 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_3_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_3_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_True_False): 0.01s I0421 03:58:21.910453 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_True): 0.01s I0421 03:58:21.917531 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_1_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_5_True_False_False_False): 0.01s I0421 03:58:21.924885 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_5_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_1_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_True_False): 0.01s I0421 03:58:21.932092 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_True): 0.01s I0421 03:58:21.939179 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_1_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_1_False_True_False_False): 0.01s I0421 03:58:21.951910 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_1_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_1_1_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_True_False): 0.01s I0421 03:58:21.964926 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_True): 0.01s I0421 03:58:21.974254 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_5_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_5_3_False_False_False_False): 0.01s I0421 03:58:21.987057 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_5_3_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_5_3_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_True_False): 0.01s I0421 03:58:22.000413 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_True): 0.01s I0421 03:58:22.013616 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_3_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_3_3_True_False_False_False): 0.02s I0421 03:58:22.030157 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_3_3_True_False_False_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_3_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_True_False): 0.01s I0421 03:58:22.039544 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_True): 0.01s I0421 03:58:22.053125 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_1_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_5_False_True_False_False): 0.01s I0421 03:58:22.066077 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_5_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_1_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_True_False): 0.01s I0421 03:58:22.079046 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_True): 0.01s I0421 03:58:22.088401 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_1_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_1_1_False_False_False_False): 0.01s I0421 03:58:22.095571 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_1_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_1_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_True_False): 0.01s I0421 03:58:22.103164 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_True): 0.01s I0421 03:58:22.110165 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_5_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_1_True_False_False_False): 0.01s I0421 03:58:22.117250 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_5_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_True_False): 0.01s I0421 03:58:22.124531 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_True): 0.01s I0421 03:58:22.131697 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_3_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_3_3_False_True_False_False): 0.01s I0421 03:58:22.138825 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_3_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_3_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_True_False): 0.01s I0421 03:58:22.145908 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_True): 0.01s I0421 03:58:22.152922 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_1_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_1_5_False_False_False_False): 0.01s I0421 03:58:22.159931 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_1_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_1_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_True_False): 0.01s I0421 03:58:22.167708 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_True): 0.01s I0421 03:58:22.174857 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_5_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_5_True_False_False_False): 0.01s I0421 03:58:22.182006 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_5_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_5_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_True_False): 0.01s I0421 03:58:22.192194 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_True): 0.01s I0421 03:58:22.199240 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_5_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_1_False_True_False_False): 0.01s I0421 03:58:22.206187 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_1_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_5_1_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_True_False): 0.01s I0421 03:58:22.213174 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_True): 0.01s I0421 03:58:22.220230 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_3_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_3_3_False_False_False_False): 0.01s I0421 03:58:22.227273 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_3_3_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_3_3_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_True_False): 0.01s I0421 03:58:22.234831 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_True): 0.01s I0421 03:58:22.241789 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_1_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_1_3_True_False_False_False): 0.01s I0421 03:58:22.248846 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_1_3_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_1_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_True_False): 0.01s I0421 03:58:22.255872 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_True): 0.01s I0421 03:58:22.262924 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_5_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_5_False_True_False_False): 0.01s I0421 03:58:22.270198 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_5_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_5_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_True_False): 0.01s I0421 03:58:22.280405 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_True): 0.01s I0421 03:58:22.287582 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_5_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_5_1_False_False_False_False): 0.01s I0421 03:58:22.294493 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_5_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_5_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_True_False): 0.01s I0421 03:58:22.302120 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_True): 0.01s I0421 03:58:22.309158 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_3_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_1_True_False_False_False): 0.01s I0421 03:58:22.316704 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_3_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_True_False): 0.01s I0421 03:58:22.324013 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_True): 0.01s I0421 03:58:22.331228 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_1_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_1_3_False_True_False_False): 0.01s I0421 03:58:22.338565 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_1_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_1_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_True_False): 0.03s I0421 03:58:22.367121 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_True_False): 0.03s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_True): 0.01s I0421 03:58:22.374004 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_5_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_5_5_False_False_False_False): 0.01s I0421 03:58:22.380957 281473367569248 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_5_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_5_5_False_False_False_False ====================================================================== FAIL: testMismatchedShape (__main__.MatMulInfixOperatorTest) MatMulInfixOperatorTest.testMismatchedShape ---------------------------------------------------------------------- tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped____MklMatMul_device_/job:localhost/replica:0/task:0/device:CPU:0}} lhs and rhs has different ndims: [3] vs. [2,2] [Op:MatMul] name: During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py", line 218, in testMismatchedShape infix_matmul( AssertionError: "(In\[0\] and In\[1\] has different ndims|In\[0\] ndims must be >= 2|Shape must be rank 2 but is rank 1)" does not match "{{function_node __wrapped____MklMatMul_device_/job:localhost/replica:0/task:0/device:CPU:0}} lhs and rhs has different ndims: [3] vs. [2,2] [Op:MatMul] name: " ---------------------------------------------------------------------- Ran 195 tests in 9.993s FAILED (failures=1, skipped=48) ================================================================================ ==================== Test output for //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu (shard 9 of 20): 2023-04-21 03:58:23.966745: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:266: ComplexWarning: Casting complex values to real discards the imaginary part a_np.imag = np.random.normal(-5, 5, /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:270: ComplexWarning: Casting complex values to real discards the imaginary part b_np.imag = np.random.normal(-5, 5, Running tests under Python 3.8.10: /usr/local/bin/python3 [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_True_False WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:153: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.config.list_physical_devices('GPU')` instead. W0421 03:58:25.952361 281473239708512 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:153: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.config.list_physical_devices('GPU')` instead. INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_True_False): 0.01s I0421 03:58:25.958114 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_True_False): 0.01s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_True): 0.0s I0421 03:58:25.959057 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_True_False_False): 0.0s I0421 03:58:25.959688 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_True_False): 0.0s I0421 03:58:25.960281 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_True): 0.0s I0421 03:58:25.960874 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_1_False_False_False_False): 0.0s I0421 03:58:25.961453 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_1_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_True_False): 0.0s I0421 03:58:25.962030 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_True): 0.0s I0421 03:58:25.962605 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_True_False_False_False): 0.0s I0421 03:58:25.963181 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_True_False): 0.0s I0421 03:58:25.963756 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_True): 0.0s I0421 03:58:25.964329 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_True_False_False): 0.0s I0421 03:58:25.964901 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_True_False 2023-04-21 03:58:26.035890: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:375] MLIR V1 optimization pass is not enabled INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_True_False): 0.3s I0421 03:58:26.268091 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_True_False): 0.3s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_True): 0.16s I0421 03:58:26.427046 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_True): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_5_False_False_False_False): 0.18s I0421 03:58:26.605581 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_5_False_False_False_False): 0.18s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_True_False): 0.18s I0421 03:58:26.788536 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_True_False): 0.18s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_True): 0.11s I0421 03:58:26.902135 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_True): 0.11s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_True_False_False_False): 0.17s I0421 03:58:27.072256 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_True_False_False_False): 0.17s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_True_False): 0.27s I0421 03:58:27.347041 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_True_False): 0.27s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_True): 0.21s I0421 03:58:27.557901 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_True): 0.21s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_True_False_False): 0.12s I0421 03:58:27.682209 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_True_False_False): 0.12s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_True_False): 0.27s I0421 03:58:27.954880 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_True_False): 0.27s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_True): 0.22s I0421 03:58:28.179154 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_True): 0.22s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_3_False_False_False_False): 0.27s I0421 03:58:28.454674 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_3_False_False_False_False): 0.27s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_3_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_True_False): 0.12s I0421 03:58:28.572679 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_True_False): 0.12s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_True): 0.11s I0421 03:58:28.681019 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_True): 0.11s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_True_False_False_False): 0.13s I0421 03:58:28.815057 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_True_False_False_False): 0.13s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_True_False): 0.14s I0421 03:58:28.958206 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_True_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_True): 0.1s I0421 03:58:29.058473 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_True): 0.1s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_True_False_False): 0.14s I0421 03:58:29.203798 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_True_False_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_True_False): 0.21s I0421 03:58:29.409972 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_True_False): 0.21s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_True): 0.21s I0421 03:58:29.625450 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_True): 0.21s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_1_False_False_False_False): 0.13s I0421 03:58:29.754407 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_1_False_False_False_False): 0.13s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_True_False): 0.21s I0421 03:58:29.964238 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_True_False): 0.21s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_True): 0.21s I0421 03:58:30.177972 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_True): 0.21s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_True_False_False_False): 0.27s I0421 03:58:30.447075 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_True_False_False_False): 0.27s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_True_False): 0.36s I0421 03:58:30.812930 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_True_False): 0.36s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_True): 0.0s I0421 03:58:30.813831 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_True_False_False): 0.0s I0421 03:58:30.814394 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_True_False): 0.0s I0421 03:58:30.814931 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_True): 0.0s I0421 03:58:30.815459 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_5_False_False_False_False): 0.0s I0421 03:58:30.815978 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_5_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_True_False): 0.0s I0421 03:58:30.816528 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_True): 0.0s I0421 03:58:30.817046 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_True_False_False_False): 0.0s I0421 03:58:30.817561 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_True_False): 0.0s I0421 03:58:30.818077 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_True): 0.0s I0421 03:58:30.818600 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_True_False_False): 0.0s I0421 03:58:30.819112 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_True_False): 0.0s I0421 03:58:30.819626 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_True): 0.15s I0421 03:58:30.968303 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_True): 0.15s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_3_False_False_False_False): 0.14s I0421 03:58:31.113152 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_3_False_False_False_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_3_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_True_False): 0.14s I0421 03:58:31.250381 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_True_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_True): 0.18s I0421 03:58:31.428599 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_True): 0.18s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_True_False_False_False): 0.14s I0421 03:58:31.567336 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_True_False_False_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_True_False): 0.14s I0421 03:58:31.704192 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_True_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_True): 0.17s I0421 03:58:31.879234 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_True): 0.17s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_True_False_False): 0.24s I0421 03:58:32.117645 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_True_False_False): 0.24s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_True_False): 0.15s I0421 03:58:32.272827 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_True_False): 0.15s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_True): 0.22s I0421 03:58:32.495619 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_True): 0.22s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_1_False_False_False_False): 0.34s I0421 03:58:32.837293 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_1_False_False_False_False): 0.34s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_True_False): 0.26s I0421 03:58:33.103103 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_True_False): 0.26s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_True): 0.09s I0421 03:58:33.193337 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_True): 0.09s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_True_False_False_False): 0.09s I0421 03:58:33.284029 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_True_False_False_False): 0.09s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_True_False): 0.11s I0421 03:58:33.395118 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_True_False): 0.11s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_True): 0.19s I0421 03:58:33.585953 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_True): 0.19s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_True_False_False): 0.1s I0421 03:58:33.688988 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_True_False_False): 0.1s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_True_False): 0.11s I0421 03:58:33.796800 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_True_False): 0.11s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_True): 0.14s I0421 03:58:33.936689 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_True): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_5_False_False_False_False): 0.22s I0421 03:58:34.161309 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_5_False_False_False_False): 0.22s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_True_False): 0.12s I0421 03:58:34.283520 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_True_False): 0.12s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_True): 0.13s I0421 03:58:34.418928 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_True): 0.13s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_True_False_False_False): 0.17s I0421 03:58:34.588651 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_True_False_False_False): 0.17s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_True_False): 0.19s I0421 03:58:34.782537 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_True_False): 0.19s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_True): 0.0s I0421 03:58:34.783639 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_True_False_False): 0.0s I0421 03:58:34.784256 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_True_False): 0.0s I0421 03:58:34.784792 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_True): 0.0s I0421 03:58:34.785314 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_3_False_False_False_False): 0.0s I0421 03:58:34.785833 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_3_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_3_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_True_False): 0.0s I0421 03:58:34.786387 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_True): 0.0s I0421 03:58:34.786908 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_True_False_False_False): 0.0s I0421 03:58:34.787433 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_True_False): 0.0s I0421 03:58:34.787945 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_True): 0.0s I0421 03:58:34.788458 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_True_False_False): 0.0s I0421 03:58:34.788974 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_True_False): 0.0s I0421 03:58:34.789484 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_True): 0.0s I0421 03:58:34.790002 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_1_False_False_False_False): 0.0s I0421 03:58:34.790513 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_1_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_True_False): 0.0s I0421 03:58:34.791021 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_True): 0.0s I0421 03:58:34.791530 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_True_False_False_False): 0.0s I0421 03:58:34.792046 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_True_False): 0.0s I0421 03:58:34.792551 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_True): 0.0s I0421 03:58:34.793071 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_True_False_False): 0.0s I0421 03:58:34.793583 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_True_False): 0.0s I0421 03:58:34.794095 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_True): 0.0s I0421 03:58:34.794601 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_5_False_False_False_False): 0.0s I0421 03:58:34.795110 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_5_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_True_False): 0.0s I0421 03:58:34.795620 281473239708512 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_True_False [ RUN ] MatMulInfixOperatorTest.testMismatchedShape INFO:tensorflow:time(__main__.MatMulInfixOperatorTest.testMismatchedShape): 0.04s I0421 03:58:34.841227 281473239708512 test_util.py:2462] time(__main__.MatMulInfixOperatorTest.testMismatchedShape): 0.04s [ FAILED ] MatMulInfixOperatorTest.testMismatchedShape [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_True_False_False_False 2023-04-21 03:58:34.856011: E ./tensorflow/core/graph/mkl_graph_util.h:182] oneDNN BFloat16 support are only on platforms with AVX512. Falling back to default implementation if present. /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:103: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. np_val = np.matrix(a_np_) * np.matrix(b_np_) /usr/local/lib/python3.8/dist-packages/numpy/matrixlib/defmatrix.py:69: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. return matrix(data, dtype=dtype, copy=False) INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_True_False_False_False): 0.02s I0421 03:58:34.864339 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_True_False_False_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_True_False): 0.01s I0421 03:58:34.875627 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_True): 0.01s I0421 03:58:34.884996 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_True_False_False): 0.01s I0421 03:58:34.894576 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_True_False): 0.01s I0421 03:58:34.904218 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_True): 0.01s I0421 03:58:34.913523 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_5_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_5_1_False_False_False_False): 0.01s I0421 03:58:34.922712 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_5_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_5_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_True_False): 0.01s I0421 03:58:34.932036 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_True): 0.01s I0421 03:58:34.940988 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_True_False_False_False): 0.01s I0421 03:58:34.956818 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_True_False): 0.01s I0421 03:58:34.966562 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_True): 0.01s I0421 03:58:34.975861 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_True_False_False): 0.02s I0421 03:58:34.994058 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_True_False_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_True_False): 0.02s I0421 03:58:35.010950 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_True): 0.01s I0421 03:58:35.027093 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_5_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_5_5_False_False_False_False): 0.01s I0421 03:58:35.036316 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_5_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_5_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_True_False): 0.01s I0421 03:58:35.051378 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_True): 0.01s I0421 03:58:35.060581 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_3_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_5_True_False_False_False): 0.02s I0421 03:58:35.079218 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_5_True_False_False_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_3_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_True_False): 0.02s I0421 03:58:35.096140 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_True): 0.01s I0421 03:58:35.105813 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_True_False_False): 0.01s I0421 03:58:35.115232 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_True_False): 0.02s I0421 03:58:35.131940 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_True): 0.01s I0421 03:58:35.147965 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_1_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_1_3_False_False_False_False): 0.01s I0421 03:58:35.158319 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_1_3_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_1_3_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_True_False): 0.01s I0421 03:58:35.173674 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_True): 0.01s I0421 03:58:35.183808 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_5_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_5_3_True_False_False_False): 0.01s I0421 03:58:35.200067 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_5_3_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_5_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_True_False): 0.02s I0421 03:58:35.217172 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_True): 0.01s I0421 03:58:35.226572 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_True_False_False): 0.01s I0421 03:58:35.235943 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_True_False): 0.01s I0421 03:58:35.251650 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_True): 0.01s I0421 03:58:35.267295 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_3_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_3_1_False_False_False_False): 0.01s I0421 03:58:35.276381 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_3_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_3_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_True_False): 0.02s I0421 03:58:35.293475 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_True): 0.01s I0421 03:58:35.302628 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_1_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_1_True_False_False_False): 0.01s I0421 03:58:35.318344 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_1_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_True_False): 0.01s I0421 03:58:35.327550 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_True): 0.01s I0421 03:58:35.335390 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_5_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_5_3_False_True_False_False): 0.01s I0421 03:58:35.343337 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_5_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_5_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_True_False): 0.01s I0421 03:58:35.351442 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_True): 0.01s I0421 03:58:35.359258 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_3_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_3_5_False_False_False_False): 0.01s I0421 03:58:35.367337 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_3_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_3_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_True_False): 0.01s I0421 03:58:35.375665 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_True): 0.01s I0421 03:58:35.383232 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_1_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_5_True_False_False_False): 0.01s I0421 03:58:35.391153 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_5_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_1_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_True_False): 0.01s I0421 03:58:35.399337 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_True): 0.01s I0421 03:58:35.407458 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_1_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_1_False_True_False_False): 0.02s I0421 03:58:35.425664 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_1_False_True_False_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_1_1_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_True_False): 0.02s I0421 03:58:35.443284 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_True): 0.01s I0421 03:58:35.456705 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_5_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_5_3_False_False_False_False): 0.01s I0421 03:58:35.470557 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_5_3_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_5_3_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_True_False): 0.01s I0421 03:58:35.485418 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_True): 0.01s I0421 03:58:35.495965 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_3_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_3_3_True_False_False_False): 0.02s I0421 03:58:35.513376 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_3_3_True_False_False_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_3_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_True_False): 0.01s I0421 03:58:35.525196 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_True): 0.01s I0421 03:58:35.539515 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_1_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_5_False_True_False_False): 0.01s I0421 03:58:35.553431 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_5_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_1_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_True_False): 0.01s I0421 03:58:35.566933 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_True): 0.01s I0421 03:58:35.577196 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_1_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_1_1_False_False_False_False): 0.01s I0421 03:58:35.584940 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_1_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_1_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_True_False): 0.01s I0421 03:58:35.599874 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_True): 0.01s I0421 03:58:35.610852 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_5_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_1_True_False_False_False): 0.01s I0421 03:58:35.618454 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_5_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_True_False): 0.01s I0421 03:58:35.625673 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_True): 0.01s I0421 03:58:35.633645 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_3_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_3_3_False_True_False_False): 0.01s I0421 03:58:35.641476 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_3_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_3_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_True_False): 0.01s I0421 03:58:35.649384 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_True): 0.01s I0421 03:58:35.657127 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_1_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_1_5_False_False_False_False): 0.01s I0421 03:58:35.664648 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_1_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_1_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_True_False): 0.01s I0421 03:58:35.673028 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_True): 0.01s I0421 03:58:35.680714 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_5_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_5_True_False_False_False): 0.01s I0421 03:58:35.688466 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_5_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_5_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_True_False): 0.01s I0421 03:58:35.699702 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_True): 0.01s I0421 03:58:35.707951 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_5_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_1_False_True_False_False): 0.01s I0421 03:58:35.715906 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_1_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_5_1_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_True_False): 0.02s I0421 03:58:35.737513 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_True): 0.01s I0421 03:58:35.745809 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_3_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_3_3_False_False_False_False): 0.01s I0421 03:58:35.754201 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_3_3_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_3_3_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_True_False): 0.01s I0421 03:58:35.763055 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_True): 0.01s I0421 03:58:35.770687 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_1_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_1_3_True_False_False_False): 0.01s I0421 03:58:35.778666 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_1_3_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_1_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_True_False): 0.01s I0421 03:58:35.786339 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_True): 0.01s I0421 03:58:35.794276 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_5_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_5_False_True_False_False): 0.01s I0421 03:58:35.802131 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_5_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_5_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_True_False): 0.01s I0421 03:58:35.812566 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_True): 0.01s I0421 03:58:35.820519 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_5_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_5_1_False_False_False_False): 0.01s I0421 03:58:35.828417 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_5_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_5_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_True_False): 0.01s I0421 03:58:35.837460 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_True): 0.01s I0421 03:58:35.845455 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_3_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_1_True_False_False_False): 0.01s I0421 03:58:35.853872 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_3_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_True_False): 0.01s I0421 03:58:35.861933 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_True): 0.01s I0421 03:58:35.870221 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_1_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_1_3_False_True_False_False): 0.01s I0421 03:58:35.878901 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_1_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_1_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_True_False): 0.01s I0421 03:58:35.893219 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_True): 0.01s I0421 03:58:35.902637 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_5_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_5_5_False_False_False_False): 0.01s I0421 03:58:35.911926 281473239708512 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_5_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_5_5_False_False_False_False ====================================================================== FAIL: testMismatchedShape (__main__.MatMulInfixOperatorTest) MatMulInfixOperatorTest.testMismatchedShape ---------------------------------------------------------------------- tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped____MklMatMul_device_/job:localhost/replica:0/task:0/device:CPU:0}} lhs and rhs has different ndims: [3] vs. [2,2] [Op:MatMul] name: During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py", line 218, in testMismatchedShape infix_matmul( AssertionError: "(In\[0\] and In\[1\] has different ndims|In\[0\] ndims must be >= 2|Shape must be rank 2 but is rank 1)" does not match "{{function_node __wrapped____MklMatMul_device_/job:localhost/replica:0/task:0/device:CPU:0}} lhs and rhs has different ndims: [3] vs. [2,2] [Op:MatMul] name: " ---------------------------------------------------------------------- Ran 195 tests in 9.961s FAILED (failures=1, skipped=48) ================================================================================ ==================== Test output for //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu (shard 9 of 20): 2023-04-21 03:58:37.396252: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:266: ComplexWarning: Casting complex values to real discards the imaginary part a_np.imag = np.random.normal(-5, 5, /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:270: ComplexWarning: Casting complex values to real discards the imaginary part b_np.imag = np.random.normal(-5, 5, Running tests under Python 3.8.10: /usr/local/bin/python3 [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_True_False WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:153: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.config.list_physical_devices('GPU')` instead. W0421 03:58:39.533497 281473039037280 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:153: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.config.list_physical_devices('GPU')` instead. INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_True_False): 0.01s I0421 03:58:39.539530 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_True_False): 0.01s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_True): 0.0s I0421 03:58:39.540480 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_True_False_False): 0.0s I0421 03:58:39.541119 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_True_False): 0.0s I0421 03:58:39.541726 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_True): 0.0s I0421 03:58:39.542332 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_1_False_False_False_False): 0.0s I0421 03:58:39.542923 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_1_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_True_False): 0.0s I0421 03:58:39.543510 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_True): 0.0s I0421 03:58:39.544095 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_True_False_False_False): 0.0s I0421 03:58:39.544682 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_True_False): 0.0s I0421 03:58:39.545273 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_True): 0.0s I0421 03:58:39.545862 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_True_False_False): 0.0s I0421 03:58:39.546478 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_True_False 2023-04-21 03:58:39.723967: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:375] MLIR V1 optimization pass is not enabled INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_True_False): 0.55s I0421 03:58:40.100088 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_True_False): 0.55s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_True): 0.18s I0421 03:58:40.281998 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_True): 0.18s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_5_False_False_False_False): 0.19s I0421 03:58:40.477378 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_5_False_False_False_False): 0.19s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_True_False): 0.21s I0421 03:58:40.685496 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_True_False): 0.21s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_True): 0.11s I0421 03:58:40.795492 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_True): 0.11s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_True_False_False_False): 0.16s I0421 03:58:40.960033 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_True_False_False_False): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_True_False): 0.2s I0421 03:58:41.160426 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_True_False): 0.2s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_True): 0.21s I0421 03:58:41.367051 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_True): 0.21s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_True_False_False): 0.12s I0421 03:58:41.484724 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_True_False_False): 0.12s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_True_False): 0.2s I0421 03:58:41.688916 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_True_False): 0.2s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_True): 0.25s I0421 03:58:41.935575 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_True): 0.25s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_3_False_False_False_False): 0.27s I0421 03:58:42.204591 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_3_False_False_False_False): 0.27s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_3_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_True_False): 0.11s I0421 03:58:42.319276 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_True_False): 0.11s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_True): 0.12s I0421 03:58:42.439132 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_True): 0.12s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_True_False_False_False): 0.15s I0421 03:58:42.593938 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_True_False_False_False): 0.15s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_True_False): 0.14s I0421 03:58:42.736241 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_True_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_True): 0.11s I0421 03:58:42.847412 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_True): 0.11s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_True_False_False): 0.14s I0421 03:58:42.992645 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_True_False_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_True_False): 0.21s I0421 03:58:43.204434 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_True_False): 0.21s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_True): 0.19s I0421 03:58:43.394788 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_True): 0.19s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_1_False_False_False_False): 0.14s I0421 03:58:43.539537 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_1_False_False_False_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_True_False): 0.2s I0421 03:58:43.742563 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_True_False): 0.2s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_True): 0.21s I0421 03:58:43.956970 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_True): 0.21s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_True_False_False_False): 0.27s I0421 03:58:44.224398 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_True_False_False_False): 0.27s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_True_False): 0.35s I0421 03:58:44.571906 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_True_False): 0.35s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_True): 0.0s I0421 03:58:44.572792 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_True_False_False): 0.0s I0421 03:58:44.573365 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_True_False): 0.0s I0421 03:58:44.573904 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_True): 0.0s I0421 03:58:44.574433 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_5_False_False_False_False): 0.0s I0421 03:58:44.574959 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_5_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_True_False): 0.0s I0421 03:58:44.575485 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_True): 0.0s I0421 03:58:44.576010 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_True_False_False_False): 0.0s I0421 03:58:44.576571 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_True_False): 0.0s I0421 03:58:44.577092 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_True): 0.0s I0421 03:58:44.577623 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_True_False_False): 0.0s I0421 03:58:44.578145 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_True_False): 0.0s I0421 03:58:44.578664 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_True): 0.17s I0421 03:58:44.748540 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_True): 0.17s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_3_False_False_False_False): 0.19s I0421 03:58:44.935421 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_3_False_False_False_False): 0.19s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_3_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_True_False): 0.16s I0421 03:58:45.091941 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_True_False): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_True): 0.21s I0421 03:58:45.297824 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_True): 0.21s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_True_False_False_False): 0.14s I0421 03:58:45.438709 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_True_False_False_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_True_False): 0.13s I0421 03:58:45.574219 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_True_False): 0.13s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_True): 0.19s I0421 03:58:45.760188 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_True): 0.19s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_True_False_False): 0.29s I0421 03:58:46.053228 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_True_False_False): 0.29s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_True_False): 0.14s I0421 03:58:46.194122 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_True_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_True): 0.16s I0421 03:58:46.354461 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_True): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_1_False_False_False_False): 0.18s I0421 03:58:46.534410 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_1_False_False_False_False): 0.18s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_True_False): 0.25s I0421 03:58:46.782856 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_True_False): 0.25s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_True): 0.09s I0421 03:58:46.870746 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_True): 0.09s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_True_False_False_False): 0.08s I0421 03:58:46.954394 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_True_False_False_False): 0.08s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_True_False): 0.11s I0421 03:58:47.061013 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_True_False): 0.11s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_True): 0.13s I0421 03:58:47.192693 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_True): 0.13s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_True_False_False): 0.1s I0421 03:58:47.289180 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_True_False_False): 0.1s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_True_False): 0.11s I0421 03:58:47.396484 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_True_False): 0.11s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_True): 0.13s I0421 03:58:47.528078 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_True): 0.13s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_5_False_False_False_False): 0.16s I0421 03:58:47.690671 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_5_False_False_False_False): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_True_False): 0.11s I0421 03:58:47.798140 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_True_False): 0.11s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_True): 0.12s I0421 03:58:47.916678 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_True): 0.12s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_True_False_False_False): 0.16s I0421 03:58:48.077802 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_True_False_False_False): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_True_False): 0.19s I0421 03:58:48.267018 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_True_False): 0.19s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_True): 0.0s I0421 03:58:48.267896 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_True_False_False): 0.0s I0421 03:58:48.268459 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_True_False): 0.0s I0421 03:58:48.268991 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_True): 0.0s I0421 03:58:48.269515 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_3_False_False_False_False): 0.0s I0421 03:58:48.270033 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_3_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_3_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_True_False): 0.0s I0421 03:58:48.270550 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_True): 0.0s I0421 03:58:48.271067 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_True_False_False_False): 0.0s I0421 03:58:48.271584 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_True_False): 0.0s I0421 03:58:48.272097 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_True): 0.0s I0421 03:58:48.272606 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_True_False_False): 0.0s I0421 03:58:48.273118 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_True_False): 0.0s I0421 03:58:48.273630 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_True): 0.0s I0421 03:58:48.274143 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_1_False_False_False_False): 0.0s I0421 03:58:48.274653 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_1_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_True_False): 0.0s I0421 03:58:48.275164 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_True): 0.0s I0421 03:58:48.275674 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_True_False_False_False): 0.0s I0421 03:58:48.276322 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_True_False): 0.0s I0421 03:58:48.276851 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_True): 0.0s I0421 03:58:48.277374 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_True_False_False): 0.0s I0421 03:58:48.277891 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_True_False): 0.0s I0421 03:58:48.278408 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_True): 0.0s I0421 03:58:48.278924 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_5_False_False_False_False): 0.0s I0421 03:58:48.279438 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_5_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_True_False): 0.0s I0421 03:58:48.279954 281473039037280 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_True_False [ RUN ] MatMulInfixOperatorTest.testMismatchedShape INFO:tensorflow:time(__main__.MatMulInfixOperatorTest.testMismatchedShape): 0.03s I0421 03:58:48.312844 281473039037280 test_util.py:2462] time(__main__.MatMulInfixOperatorTest.testMismatchedShape): 0.03s [ FAILED ] MatMulInfixOperatorTest.testMismatchedShape [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_True_False_False_False 2023-04-21 03:58:48.324607: E ./tensorflow/core/graph/mkl_graph_util.h:182] oneDNN BFloat16 support are only on platforms with AVX512. Falling back to default implementation if present. /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:103: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. np_val = np.matrix(a_np_) * np.matrix(b_np_) /usr/local/lib/python3.8/dist-packages/numpy/matrixlib/defmatrix.py:69: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. return matrix(data, dtype=dtype, copy=False) INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_True_False_False_False): 0.01s I0421 03:58:48.329393 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_True_False): 0.01s I0421 03:58:48.338152 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_True): 0.01s I0421 03:58:48.345561 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_True_False_False): 0.01s I0421 03:58:48.353051 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_True_False): 0.01s I0421 03:58:48.360753 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_True): 0.01s I0421 03:58:48.368320 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_5_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_5_1_False_False_False_False): 0.01s I0421 03:58:48.375838 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_5_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_5_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_True_False): 0.01s I0421 03:58:48.383369 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_True): 0.01s I0421 03:58:48.391009 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_True_False_False_False): 0.02s I0421 03:58:48.412760 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_True_False_False_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_True_False): 0.01s I0421 03:58:48.420650 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_True): 0.01s I0421 03:58:48.428417 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_True_False_False): 0.01s I0421 03:58:48.435486 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_True_False): 0.01s I0421 03:58:48.448559 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_True): 0.01s I0421 03:58:48.462169 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_5_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_5_5_False_False_False_False): 0.01s I0421 03:58:48.469606 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_5_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_5_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_True_False): 0.01s I0421 03:58:48.482782 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_True): 0.01s I0421 03:58:48.490531 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_3_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_5_True_False_False_False): 0.01s I0421 03:58:48.505237 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_5_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_3_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_True_False): 0.01s I0421 03:58:48.520486 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_True): 0.01s I0421 03:58:48.528589 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_True_False_False): 0.01s I0421 03:58:48.536436 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_True_False): 0.01s I0421 03:58:48.550351 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_True): 0.01s I0421 03:58:48.564703 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_1_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_1_3_False_False_False_False): 0.01s I0421 03:58:48.578070 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_1_3_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_1_3_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_True_False): 0.01s I0421 03:58:48.592265 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_True): 0.01s I0421 03:58:48.600850 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_5_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_5_3_True_False_False_False): 0.01s I0421 03:58:48.614662 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_5_3_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_5_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_True_False): 0.01s I0421 03:58:48.629296 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_True): 0.01s I0421 03:58:48.637046 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_True_False_False): 0.01s I0421 03:58:48.645222 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_True_False): 0.01s I0421 03:58:48.659954 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_True): 0.01s I0421 03:58:48.674559 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_3_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_3_1_False_False_False_False): 0.01s I0421 03:58:48.682708 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_3_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_3_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_True_False): 0.02s I0421 03:58:48.698890 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_True): 0.01s I0421 03:58:48.707242 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_1_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_1_True_False_False_False): 0.01s I0421 03:58:48.718762 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_1_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_True_False): 0.01s I0421 03:58:48.726324 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_True): 0.01s I0421 03:58:48.733767 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_5_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_5_3_False_True_False_False): 0.01s I0421 03:58:48.741423 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_5_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_5_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_True_False): 0.01s I0421 03:58:48.749569 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_True): 0.01s I0421 03:58:48.757396 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_3_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_3_5_False_False_False_False): 0.01s I0421 03:58:48.764953 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_3_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_3_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_True_False): 0.01s I0421 03:58:48.773602 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_True): 0.01s I0421 03:58:48.781414 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_1_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_5_True_False_False_False): 0.01s I0421 03:58:48.789327 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_5_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_1_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_True_False): 0.01s I0421 03:58:48.797947 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_True): 0.01s I0421 03:58:48.805440 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_1_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_1_False_True_False_False): 0.02s I0421 03:58:48.823627 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_1_False_True_False_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_1_1_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_True_False): 0.02s I0421 03:58:48.840671 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_True): 0.01s I0421 03:58:48.854836 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_5_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_5_3_False_False_False_False): 0.02s I0421 03:58:48.872208 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_5_3_False_False_False_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_5_3_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_True_False): 0.01s I0421 03:58:48.886519 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_True): 0.01s I0421 03:58:48.896419 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_3_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_3_3_True_False_False_False): 0.02s I0421 03:58:48.915571 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_3_3_True_False_False_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_3_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_True_False): 0.01s I0421 03:58:48.927780 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_True): 0.01s I0421 03:58:48.943034 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_1_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_5_False_True_False_False): 0.02s I0421 03:58:48.961949 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_5_False_True_False_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_1_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_True_False): 0.01s I0421 03:58:48.977787 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_True): 0.01s I0421 03:58:48.993253 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_1_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_1_1_False_False_False_False): 0.01s I0421 03:58:49.001623 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_1_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_1_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_True_False): 0.01s I0421 03:58:49.010518 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_True): 0.01s I0421 03:58:49.018085 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_5_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_1_True_False_False_False): 0.01s I0421 03:58:49.025470 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_5_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_True_False): 0.01s I0421 03:58:49.032807 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_True): 0.01s I0421 03:58:49.040208 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_3_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_3_3_False_True_False_False): 0.01s I0421 03:58:49.047404 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_3_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_3_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_True_False): 0.01s I0421 03:58:49.054834 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_True): 0.01s I0421 03:58:49.062591 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_1_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_1_5_False_False_False_False): 0.01s I0421 03:58:49.069747 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_1_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_1_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_True_False): 0.01s I0421 03:58:49.078076 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_True): 0.01s I0421 03:58:49.085569 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_5_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_5_True_False_False_False): 0.01s I0421 03:58:49.093770 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_5_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_5_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_True_False): 0.01s I0421 03:58:49.104099 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_True): 0.01s I0421 03:58:49.111683 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_5_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_1_False_True_False_False): 0.01s I0421 03:58:49.119403 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_1_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_5_1_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_True_False): 0.01s I0421 03:58:49.127079 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_True): 0.01s I0421 03:58:49.134656 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_3_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_3_3_False_False_False_False): 0.01s I0421 03:58:49.142407 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_3_3_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_3_3_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_True_False): 0.01s I0421 03:58:49.150885 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_True): 0.01s I0421 03:58:49.158164 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_1_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_1_3_True_False_False_False): 0.01s I0421 03:58:49.165396 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_1_3_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_1_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_True_False): 0.01s I0421 03:58:49.172656 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_True): 0.01s I0421 03:58:49.179734 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_5_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_5_False_True_False_False): 0.01s I0421 03:58:49.187271 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_5_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_5_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_True_False): 0.01s I0421 03:58:49.197061 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_True): 0.01s I0421 03:58:49.205156 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_5_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_5_1_False_False_False_False): 0.01s I0421 03:58:49.212727 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_5_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_5_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_True_False): 0.01s I0421 03:58:49.220978 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_True): 0.01s I0421 03:58:49.228681 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_3_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_1_True_False_False_False): 0.01s I0421 03:58:49.236221 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_3_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_True_False): 0.01s I0421 03:58:49.243569 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_True): 0.01s I0421 03:58:49.251187 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_1_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_1_3_False_True_False_False): 0.01s I0421 03:58:49.258473 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_1_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_1_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_True_False): 0.01s I0421 03:58:49.267260 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_True): 0.01s I0421 03:58:49.274857 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_5_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_5_5_False_False_False_False): 0.01s I0421 03:58:49.282366 281473039037280 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_5_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_5_5_False_False_False_False ====================================================================== FAIL: testMismatchedShape (__main__.MatMulInfixOperatorTest) MatMulInfixOperatorTest.testMismatchedShape ---------------------------------------------------------------------- tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped____MklMatMul_device_/job:localhost/replica:0/task:0/device:CPU:0}} lhs and rhs has different ndims: [3] vs. [2,2] [Op:MatMul] name: During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py", line 218, in testMismatchedShape infix_matmul( AssertionError: "(In\[0\] and In\[1\] has different ndims|In\[0\] ndims must be >= 2|Shape must be rank 2 but is rank 1)" does not match "{{function_node __wrapped____MklMatMul_device_/job:localhost/replica:0/task:0/device:CPU:0}} lhs and rhs has different ndims: [3] vs. [2,2] [Op:MatMul] name: " ---------------------------------------------------------------------- Ran 195 tests in 9.750s FAILED (failures=1, skipped=48) ================================================================================ ==================== Test output for //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu (shard 8 of 20): 2023-04-21 03:58:07.341086: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:266: ComplexWarning: Casting complex values to real discards the imaginary part a_np.imag = np.random.normal(-5, 5, /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:270: ComplexWarning: Casting complex values to real discards the imaginary part b_np.imag = np.random.normal(-5, 5, Running tests under Python 3.8.10: /usr/local/bin/python3 [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_False_True WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:153: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.config.list_physical_devices('GPU')` instead. W0421 03:58:12.457795 281472934769504 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:153: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.config.list_physical_devices('GPU')` instead. INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_False_True): 0.01s I0421 03:58:12.463577 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_False_True): 0.01s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_False): 0.0s I0421 03:58:12.464467 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_False_True_False): 0.0s I0421 03:58:12.465110 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_False_True): 0.0s I0421 03:58:12.465710 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_False): 0.0s I0421 03:58:12.466333 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_5_True_False_True_False): 0.0s I0421 03:58:12.466932 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_5_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_False_True): 0.0s I0421 03:58:12.467515 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_False): 0.0s I0421 03:58:12.468094 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_False_True_True_False): 0.0s I0421 03:58:12.468671 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_False_True): 0.0s I0421 03:58:12.469247 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_False): 0.0s I0421 03:58:12.469820 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_False_True_False): 0.0s I0421 03:58:12.470394 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_False_True 2023-04-21 03:58:12.508463: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:375] MLIR V1 optimization pass is not enabled INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_False_True): 0.14s I0421 03:58:12.608430 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_False_True): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_False): 0.12s I0421 03:58:12.727957 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_False): 0.12s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_3_True_False_True_False): 0.15s I0421 03:58:12.880937 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_3_True_False_True_False): 0.15s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_False_True): 0.15s I0421 03:58:13.029880 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_False_True): 0.15s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_False): 0.1s I0421 03:58:13.125820 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_False): 0.1s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_False_True_True_False): 0.16s I0421 03:58:13.281675 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_False_True_True_False): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_False_True): 0.2s I0421 03:58:13.485814 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_False_True): 0.2s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_False): 0.22s I0421 03:58:13.710822 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_False): 0.22s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_False_True_False): 0.16s I0421 03:58:13.870547 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_False_True_False): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_False_True): 0.22s I0421 03:58:14.094192 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_False_True): 0.22s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_False): 0.27s I0421 03:58:14.367770 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_False): 0.27s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_1_True_False_True_False): 0.39s I0421 03:58:14.760683 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_1_True_False_True_False): 0.39s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_False_True): 0.09s I0421 03:58:14.849366 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_False_True): 0.09s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_False): 0.1s I0421 03:58:14.952422 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_False): 0.1s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_False_True_True_False): 0.13s I0421 03:58:15.086831 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_False_True_True_False): 0.13s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_False_True): 0.16s I0421 03:58:15.247355 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_False_True): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_False): 0.1s I0421 03:58:15.345022 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_False): 0.1s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_False_True_False): 0.21s I0421 03:58:15.552774 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_False_True_False): 0.21s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_False_True): 0.17s I0421 03:58:15.725340 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_False_True): 0.17s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_False): 0.19s I0421 03:58:15.911388 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_False): 0.19s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_5_True_False_True_False): 0.39s I0421 03:58:16.305394 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_5_True_False_True_False): 0.39s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_False_True): 0.19s I0421 03:58:16.496013 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_False_True): 0.19s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_False): 0.27s I0421 03:58:16.770300 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_False): 0.27s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_False_True_True_False): 0.25s I0421 03:58:17.022068 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_False_True_True_False): 0.25s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_False_True): 0.4s I0421 03:58:17.424628 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_False_True): 0.4s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_False): 0.0s I0421 03:58:17.425595 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_False_True_False): 0.0s I0421 03:58:17.426192 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_False_True): 0.0s I0421 03:58:17.426735 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_False): 0.0s I0421 03:58:17.427257 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_3_True_False_True_False): 0.0s I0421 03:58:17.427773 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_3_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_False_True): 0.0s I0421 03:58:17.428290 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_False): 0.0s I0421 03:58:17.428802 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_False_True_True_False): 0.0s I0421 03:58:17.429311 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_False_True): 0.0s I0421 03:58:17.429821 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_False): 0.0s I0421 03:58:17.430337 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_False_True_False): 0.0s I0421 03:58:17.430848 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_False_True): 0.0s I0421 03:58:17.431355 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_False): 0.16s I0421 03:58:17.589683 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_False): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_1_True_False_True_False): 0.13s I0421 03:58:17.719005 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_1_True_False_True_False): 0.13s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_False_True): 0.14s I0421 03:58:17.857294 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_False_True): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_False): 0.22s I0421 03:58:18.076035 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_False): 0.22s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_False_True_True_False): 0.14s I0421 03:58:18.221688 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_False_True_True_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_False_True): 0.12s I0421 03:58:18.344591 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_False_True): 0.12s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_False): 0.16s I0421 03:58:18.510272 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_False): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_False_True_False): 0.23s I0421 03:58:18.740843 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_False_True_False): 0.23s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_False_True): 0.15s I0421 03:58:18.892764 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_False_True): 0.15s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_False): 0.18s I0421 03:58:19.077402 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_False): 0.18s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_5_True_False_True_False): 0.23s I0421 03:58:19.304330 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_5_True_False_True_False): 0.23s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_False_True): 0.32s I0421 03:58:19.625502 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_False_True): 0.32s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_False): 0.08s I0421 03:58:19.710223 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_False): 0.08s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_False_True_True_False): 0.09s I0421 03:58:19.800868 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_False_True_True_False): 0.09s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_False_True): 0.1s I0421 03:58:19.903645 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_False_True): 0.1s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_False): 0.15s I0421 03:58:20.050662 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_False): 0.15s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_False_True_False): 0.14s I0421 03:58:20.189205 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_False_True_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_False_True): 0.1s I0421 03:58:20.288209 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_False_True): 0.1s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_False): 0.13s I0421 03:58:20.419727 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_False): 0.13s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_3_True_False_True_False): 0.13s I0421 03:58:20.554600 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_3_True_False_True_False): 0.13s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_False_True): 0.1s I0421 03:58:20.658821 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_False_True): 0.1s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_False): 0.13s I0421 03:58:20.793420 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_False): 0.13s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_False_True_True_False): 0.19s I0421 03:58:20.983028 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_False_True_True_False): 0.19s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_False_True): 0.16s I0421 03:58:21.145298 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_False_True): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_False): 0.0s I0421 03:58:21.146203 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_False_True_False): 0.0s I0421 03:58:21.146795 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_False_True): 0.0s I0421 03:58:21.147327 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_False): 0.0s I0421 03:58:21.147843 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_1_True_False_True_False): 0.0s I0421 03:58:21.148357 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_1_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_False_True): 0.0s I0421 03:58:21.148868 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_False): 0.0s I0421 03:58:21.149379 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_False_True_True_False): 0.0s I0421 03:58:21.149900 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_False_True): 0.0s I0421 03:58:21.150410 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_False): 0.0s I0421 03:58:21.150916 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_False_True_False): 0.0s I0421 03:58:21.151426 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_False_True): 0.0s I0421 03:58:21.151933 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_False): 0.0s I0421 03:58:21.152448 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_5_True_False_True_False): 0.0s I0421 03:58:21.152958 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_5_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_False_True): 0.0s I0421 03:58:21.153468 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_False): 0.0s I0421 03:58:21.153976 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_False_True_True_False): 0.0s I0421 03:58:21.154497 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_False_True): 0.0s I0421 03:58:21.155002 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_False): 0.0s I0421 03:58:21.155513 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_False_True_False): 0.0s I0421 03:58:21.156018 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_False_True): 0.0s I0421 03:58:21.156606 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_False): 0.0s I0421 03:58:21.157114 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_3_True_False_True_False): 0.0s I0421 03:58:21.157620 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_3_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_False_True): 0.0s I0421 03:58:21.158130 281472934769504 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_False_True [ RUN ] MatMulInfixOperatorTest.testMismatchedDimensions INFO:tensorflow:time(__main__.MatMulInfixOperatorTest.testMismatchedDimensions): 0.03s I0421 03:58:21.187725 281472934769504 test_util.py:2462] time(__main__.MatMulInfixOperatorTest.testMismatchedDimensions): 0.03s [ FAILED ] MatMulInfixOperatorTest.testMismatchedDimensions [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_False_True_True_False 2023-04-21 03:58:21.199081: E ./tensorflow/core/graph/mkl_graph_util.h:182] oneDNN BFloat16 support are only on platforms with AVX512. Falling back to default implementation if present. /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:103: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. np_val = np.matrix(a_np_) * np.matrix(b_np_) /usr/local/lib/python3.8/dist-packages/numpy/matrixlib/defmatrix.py:69: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. return matrix(data, dtype=dtype, copy=False) INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_False_True_True_False): 0.01s I0421 03:58:21.203190 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_False_True): 0.01s I0421 03:58:21.210609 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_False): 0.01s I0421 03:58:21.218308 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_False_True_False): 0.01s I0421 03:58:21.225315 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_False_True): 0.01s I0421 03:58:21.232525 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_False): 0.01s I0421 03:58:21.239531 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_3_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_3_5_True_False_True_False): 0.01s I0421 03:58:21.246526 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_3_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_3_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_False_True): 0.01s I0421 03:58:21.253650 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_False): 0.01s I0421 03:58:21.260780 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_False_True_True_False): 0.01s I0421 03:58:21.267999 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_False_True): 0.01s I0421 03:58:21.275057 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_False): 0.01s I0421 03:58:21.282719 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_False_True_False): 0.01s I0421 03:58:21.295649 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_False_True): 0.01s I0421 03:58:21.302870 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_False): 0.03s I0421 03:58:21.337105 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_False): 0.03s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_5_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_5_3_True_False_True_False): 0.01s I0421 03:58:21.351539 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_5_3_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_5_3_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_False_True): 0.02s I0421 03:58:21.377127 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_False_True): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_False): 0.01s I0421 03:58:21.384449 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_3_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_5_False_True_True_False): 0.03s I0421 03:58:21.417243 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_5_False_True_True_False): 0.03s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_3_5_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_False_True): 0.02s I0421 03:58:21.434826 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_False_True): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_False): 0.01s I0421 03:58:21.441965 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_False_True_False): 0.01s I0421 03:58:21.454745 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_False_True): 0.01s I0421 03:58:21.461970 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_False): 0.01s I0421 03:58:21.474948 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_1_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_1_1_True_False_True_False): 0.02s I0421 03:58:21.494200 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_1_1_True_False_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_1_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_False_True): 0.01s I0421 03:58:21.501645 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_False): 0.01s I0421 03:58:21.509109 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_5_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_5_3_False_True_True_False): 0.01s I0421 03:58:21.522330 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_5_3_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_5_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_False_True): 0.01s I0421 03:58:21.536281 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_False): 0.01s I0421 03:58:21.543422 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_False_True_False): 0.01s I0421 03:58:21.556397 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_False_True): 0.01s I0421 03:58:21.563523 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_False): 0.01s I0421 03:58:21.576870 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_1_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_1_5_True_False_True_False): 0.01s I0421 03:58:21.591006 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_1_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_1_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_False_True): 0.01s I0421 03:58:21.598385 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_False): 0.01s I0421 03:58:21.605517 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_1_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_1_False_True_True_False): 0.01s I0421 03:58:21.617144 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_1_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_1_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_False_True): 0.01s I0421 03:58:21.624330 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_False): 0.01s I0421 03:58:21.631294 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_5_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_5_3_False_False_True_False): 0.01s I0421 03:58:21.638423 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_5_3_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_5_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_False_True): 0.01s I0421 03:58:21.645453 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_False): 0.01s I0421 03:58:21.652467 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_3_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_3_3_True_False_True_False): 0.01s I0421 03:58:21.659786 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_3_3_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_3_3_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_False_True): 0.01s I0421 03:58:21.667287 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_False): 0.01s I0421 03:58:21.674952 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_1_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_5_False_True_True_False): 0.01s I0421 03:58:21.682662 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_5_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_1_5_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_False_True): 0.01s I0421 03:58:21.690093 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_False): 0.01s I0421 03:58:21.697035 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_1_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_1_False_False_True_False): 0.01s I0421 03:58:21.709851 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_1_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_1_1_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_False_True): 0.01s I0421 03:58:21.722712 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_False): 0.02s I0421 03:58:21.738872 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_5_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_5_1_True_False_True_False): 0.01s I0421 03:58:21.748200 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_5_1_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_5_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_False_True): 0.01s I0421 03:58:21.757455 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_False): 0.01s I0421 03:58:21.770192 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_3_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_3_3_False_True_True_False): 0.01s I0421 03:58:21.784499 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_3_3_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_3_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_False_True): 0.01s I0421 03:58:21.795576 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_False): 0.01s I0421 03:58:21.808538 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_1_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_5_False_False_True_False): 0.01s I0421 03:58:21.818064 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_1_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_False_True): 0.01s I0421 03:58:21.830969 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_False): 0.01s I0421 03:58:21.843935 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_5_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_5_5_True_False_True_False): 0.01s I0421 03:58:21.857087 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_5_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_5_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_False_True): 0.01s I0421 03:58:21.864212 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_False): 0.01s I0421 03:58:21.871219 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_5_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_1_False_True_True_False): 0.01s I0421 03:58:21.878869 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_1_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_5_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_False_True): 0.01s I0421 03:58:21.885866 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_False): 0.01s I0421 03:58:21.892858 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_3_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_3_3_False_False_True_False): 0.01s I0421 03:58:21.899902 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_3_3_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_3_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_False_True): 0.01s I0421 03:58:21.906970 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_False): 0.01s I0421 03:58:21.914065 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_1_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_1_3_True_False_True_False): 0.01s I0421 03:58:21.927171 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_1_3_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_1_3_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_False_True): 0.01s I0421 03:58:21.934472 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_False): 0.01s I0421 03:58:21.941571 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_5_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_5_False_True_True_False): 0.01s I0421 03:58:21.949472 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_5_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_5_5_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_False_True): 0.01s I0421 03:58:21.959752 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_False): 0.01s I0421 03:58:21.967072 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_5_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_1_False_False_True_False): 0.01s I0421 03:58:21.974242 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_1_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_5_1_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_False_True): 0.01s I0421 03:58:21.981366 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_False): 0.01s I0421 03:58:21.988379 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_3_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_3_1_True_False_True_False): 0.01s I0421 03:58:21.995362 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_3_1_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_3_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_False_True): 0.01s I0421 03:58:22.002525 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_False): 0.01s I0421 03:58:22.009641 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_1_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_1_3_False_True_True_False): 0.02s I0421 03:58:22.028275 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_1_3_False_True_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_1_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_False_True): 0.01s I0421 03:58:22.035472 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_False): 0.01s I0421 03:58:22.042838 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_5_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_5_False_False_True_False): 0.01s I0421 03:58:22.049954 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_5_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_False_True): 0.01s I0421 03:58:22.060982 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_False): 0.01s I0421 03:58:22.068291 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_3_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_3_5_True_False_True_False): 0.01s I0421 03:58:22.075388 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_3_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_3_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_False_True): 0.01s I0421 03:58:22.089284 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_False): 0.01s I0421 03:58:22.096449 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_3_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_1_False_True_True_False): 0.01s I0421 03:58:22.103994 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_1_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_3_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_False_True): 0.01s I0421 03:58:22.111056 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_False): 0.01s I0421 03:58:22.118138 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_1_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_1_3_False_False_True_False): 0.01s I0421 03:58:22.125159 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_1_3_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_1_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_False_True): 0.01s I0421 03:58:22.132417 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_False): 0.01s I0421 03:58:22.142841 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_5_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_5_3_True_False_True_False): 0.01s I0421 03:58:22.150063 281472934769504 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_5_3_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_5_3_True_False_True_False ====================================================================== FAIL: testMismatchedDimensions (__main__.MatMulInfixOperatorTest) MatMulInfixOperatorTest.testMismatchedDimensions ---------------------------------------------------------------------- tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped____MklMatMul_device_/job:localhost/replica:0/task:0/device:CPU:0}} lhs mismatch rhs shape: 3 vs. 2: [1,3] [2,2] 0 0 [Op:MatMul] name: During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py", line 226, in testMismatchedDimensions infix_matmul( AssertionError: "(Matrix size-incompatible: In\[0\]: .* In\[1\]|Dimensions must be equal)" does not match "{{function_node __wrapped____MklMatMul_device_/job:localhost/replica:0/task:0/device:CPU:0}} lhs mismatch rhs shape: 3 vs. 2: [1,3] [2,2] 0 0 [Op:MatMul] name: " ---------------------------------------------------------------------- Ran 195 tests in 9.694s FAILED (failures=1, skipped=48) ================================================================================ ==================== Test output for //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu (shard 8 of 20): 2023-04-21 03:58:23.526374: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:266: ComplexWarning: Casting complex values to real discards the imaginary part a_np.imag = np.random.normal(-5, 5, /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:270: ComplexWarning: Casting complex values to real discards the imaginary part b_np.imag = np.random.normal(-5, 5, Running tests under Python 3.8.10: /usr/local/bin/python3 [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_False_True WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:153: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.config.list_physical_devices('GPU')` instead. W0421 03:58:25.599968 281473501983584 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:153: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.config.list_physical_devices('GPU')` instead. INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_False_True): 0.01s I0421 03:58:25.605453 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_False_True): 0.01s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_False): 0.0s I0421 03:58:25.606427 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_False_True_False): 0.0s I0421 03:58:25.607077 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_False_True): 0.0s I0421 03:58:25.607670 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_False): 0.0s I0421 03:58:25.608261 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_5_True_False_True_False): 0.0s I0421 03:58:25.608845 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_5_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_False_True): 0.0s I0421 03:58:25.609426 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_False): 0.0s I0421 03:58:25.610006 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_False_True_True_False): 0.0s I0421 03:58:25.610581 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_False_True): 0.0s I0421 03:58:25.611157 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_False): 0.0s I0421 03:58:25.611732 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_False_True_False): 0.0s I0421 03:58:25.612307 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_False_True 2023-04-21 03:58:25.813518: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:375] MLIR V1 optimization pass is not enabled INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_False_True): 0.33s I0421 03:58:25.945254 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_False_True): 0.33s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_False): 0.14s I0421 03:58:26.090667 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_3_True_False_True_False): 0.16s I0421 03:58:26.254080 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_3_True_False_True_False): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_False_True): 0.2s I0421 03:58:26.458421 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_False_True): 0.2s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_False): 0.14s I0421 03:58:26.594912 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_False_True_True_False): 0.19s I0421 03:58:26.782340 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_False_True_True_False): 0.19s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_False_True): 0.27s I0421 03:58:27.048682 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_False_True): 0.27s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_False): 0.25s I0421 03:58:27.299238 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_False): 0.25s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_False_True_False): 0.16s I0421 03:58:27.455475 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_False_True_False): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_False_True): 0.27s I0421 03:58:27.724479 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_False_True): 0.27s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_False): 0.34s I0421 03:58:28.061144 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_False): 0.34s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_1_True_False_True_False): 0.36s I0421 03:58:28.421808 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_1_True_False_True_False): 0.36s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_False_True): 0.1s I0421 03:58:28.520347 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_False_True): 0.1s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_False): 0.12s I0421 03:58:28.645492 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_False): 0.12s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_False_True_True_False): 0.14s I0421 03:58:28.785654 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_False_True_True_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_False_True): 0.14s I0421 03:58:28.923077 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_False_True): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_False): 0.1s I0421 03:58:29.020657 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_False): 0.1s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_False_True_False): 0.17s I0421 03:58:29.191016 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_False_True_False): 0.17s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_False_True): 0.21s I0421 03:58:29.401577 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_False_True): 0.21s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_False): 0.23s I0421 03:58:29.629357 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_False): 0.23s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_5_True_False_True_False): 0.33s I0421 03:58:29.958562 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_5_True_False_True_False): 0.33s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_False_True): 0.21s I0421 03:58:30.166074 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_False_True): 0.21s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_False): 0.25s I0421 03:58:30.419837 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_False): 0.25s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_False_True_True_False): 0.24s I0421 03:58:30.664829 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_False_True_True_False): 0.24s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_False_True): 0.45s I0421 03:58:31.119647 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_False_True): 0.45s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_False): 0.0s I0421 03:58:31.120778 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_False_True_False): 0.0s I0421 03:58:31.121398 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_False_True): 0.0s I0421 03:58:31.121944 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_False): 0.0s I0421 03:58:31.122477 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_3_True_False_True_False): 0.0s I0421 03:58:31.123007 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_3_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_False_True): 0.0s I0421 03:58:31.123532 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_False): 0.0s I0421 03:58:31.124059 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_False_True_True_False): 0.0s I0421 03:58:31.124585 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_False_True): 0.0s I0421 03:58:31.125109 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_False): 0.0s I0421 03:58:31.125638 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_False_True_False): 0.0s I0421 03:58:31.126190 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_False_True): 0.0s I0421 03:58:31.126741 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_False): 0.15s I0421 03:58:31.277060 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_False): 0.15s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_1_True_False_True_False): 0.14s I0421 03:58:31.420785 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_1_True_False_True_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_False_True): 0.15s I0421 03:58:31.573611 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_False_True): 0.15s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_False): 0.27s I0421 03:58:31.846190 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_False): 0.27s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_False_True_True_False): 0.16s I0421 03:58:32.009526 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_False_True_True_False): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_False_True): 0.13s I0421 03:58:32.137968 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_False_True): 0.13s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_False): 0.21s I0421 03:58:32.346373 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_False): 0.21s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_False_True_False): 0.39s I0421 03:58:32.732445 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_False_True_False): 0.39s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_False_True): 0.15s I0421 03:58:32.882308 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_False_True): 0.15s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_False): 0.18s I0421 03:58:33.063946 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_False): 0.18s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_5_True_False_True_False): 0.25s I0421 03:58:33.316776 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_5_True_False_True_False): 0.25s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_False_True): 0.35s I0421 03:58:33.671119 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_False_True): 0.35s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_False): 0.12s I0421 03:58:33.796505 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_False): 0.12s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_False_True_True_False): 0.1s I0421 03:58:33.894635 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_False_True_True_False): 0.1s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_False_True): 0.12s I0421 03:58:34.016505 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_False_True): 0.12s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_False): 0.14s I0421 03:58:34.153070 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_False_True_False): 0.11s I0421 03:58:34.263030 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_False_True_False): 0.11s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_False_True): 0.11s I0421 03:58:34.377924 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_False_True): 0.11s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_False): 0.19s I0421 03:58:34.568416 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_False): 0.19s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_3_True_False_True_False): 0.16s I0421 03:58:34.727089 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_3_True_False_True_False): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_False_True): 0.17s I0421 03:58:34.893516 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_False_True): 0.17s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_False): 0.21s I0421 03:58:35.101297 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_False): 0.21s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_False_True_True_False): 0.18s I0421 03:58:35.280065 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_False_True_True_False): 0.18s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_False_True): 0.18s I0421 03:58:35.464262 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_False_True): 0.18s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_False): 0.0s I0421 03:58:35.465359 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_False_True_False): 0.0s I0421 03:58:35.465955 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_False_True): 0.0s I0421 03:58:35.466540 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_False): 0.0s I0421 03:58:35.467069 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_1_True_False_True_False): 0.0s I0421 03:58:35.467595 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_1_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_False_True): 0.0s I0421 03:58:35.468119 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_False): 0.0s I0421 03:58:35.468640 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_False_True_True_False): 0.0s I0421 03:58:35.469171 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_False_True): 0.0s I0421 03:58:35.469693 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_False): 0.0s I0421 03:58:35.470207 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_False_True_False): 0.0s I0421 03:58:35.470728 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_False_True): 0.0s I0421 03:58:35.471244 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_False): 0.0s I0421 03:58:35.471767 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_5_True_False_True_False): 0.0s I0421 03:58:35.472287 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_5_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_False_True): 0.0s I0421 03:58:35.472807 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_False): 0.0s I0421 03:58:35.473324 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_False_True_True_False): 0.0s I0421 03:58:35.473854 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_False_True): 0.0s I0421 03:58:35.474375 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_False): 0.0s I0421 03:58:35.474920 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_False_True_False): 0.0s I0421 03:58:35.475445 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_False_True): 0.0s I0421 03:58:35.475968 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_False): 0.0s I0421 03:58:35.476542 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_3_True_False_True_False): 0.0s I0421 03:58:35.477066 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_3_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_False_True): 0.0s I0421 03:58:35.477592 281473501983584 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_False_True [ RUN ] MatMulInfixOperatorTest.testMismatchedDimensions INFO:tensorflow:time(__main__.MatMulInfixOperatorTest.testMismatchedDimensions): 0.04s I0421 03:58:35.519355 281473501983584 test_util.py:2462] time(__main__.MatMulInfixOperatorTest.testMismatchedDimensions): 0.04s [ FAILED ] MatMulInfixOperatorTest.testMismatchedDimensions [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_False_True_True_False 2023-04-21 03:58:35.533867: E ./tensorflow/core/graph/mkl_graph_util.h:182] oneDNN BFloat16 support are only on platforms with AVX512. Falling back to default implementation if present. /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:103: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. np_val = np.matrix(a_np_) * np.matrix(b_np_) /usr/local/lib/python3.8/dist-packages/numpy/matrixlib/defmatrix.py:69: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. return matrix(data, dtype=dtype, copy=False) INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_False_True_True_False): 0.02s I0421 03:58:35.541909 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_False_True_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_False_True): 0.01s I0421 03:58:35.551461 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_False): 0.01s I0421 03:58:35.561635 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_False_True_False): 0.01s I0421 03:58:35.570849 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_False_True): 0.01s I0421 03:58:35.580129 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_False): 0.01s I0421 03:58:35.589126 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_3_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_3_5_True_False_True_False): 0.01s I0421 03:58:35.598397 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_3_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_3_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_False_True): 0.01s I0421 03:58:35.607769 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_False): 0.01s I0421 03:58:35.616896 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_False_True_True_False): 0.01s I0421 03:58:35.625890 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_False_True): 0.01s I0421 03:58:35.634595 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_False): 0.01s I0421 03:58:35.649706 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_False_True_False): 0.02s I0421 03:58:35.665972 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_False_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_False_True): 0.01s I0421 03:58:35.674769 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_False): 0.02s I0421 03:58:35.691276 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_5_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_5_3_True_False_True_False): 0.02s I0421 03:58:35.708201 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_5_3_True_False_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_5_3_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_False_True): 0.01s I0421 03:58:35.716948 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_False): 0.01s I0421 03:58:35.726185 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_3_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_5_False_True_True_False): 0.02s I0421 03:58:35.743611 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_5_False_True_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_3_5_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_False_True): 0.02s I0421 03:58:35.761617 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_False_True): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_False): 0.01s I0421 03:58:35.770788 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_False_True_False): 0.01s I0421 03:58:35.786581 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_False_True): 0.01s I0421 03:58:35.795712 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_False): 0.02s I0421 03:58:35.812572 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_1_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_1_1_True_False_True_False): 0.02s I0421 03:58:35.829991 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_1_1_True_False_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_1_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_False_True): 0.01s I0421 03:58:35.839273 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_False): 0.01s I0421 03:58:35.848631 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_5_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_5_3_False_True_True_False): 0.02s I0421 03:58:35.865203 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_5_3_False_True_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_5_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_False_True): 0.02s I0421 03:58:35.883205 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_False_True): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_False): 0.02s I0421 03:58:35.902482 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_False_True_False): 0.02s I0421 03:58:35.921030 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_False_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_False_True): 0.01s I0421 03:58:35.931296 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_False): 0.02s I0421 03:58:35.950824 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_1_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_1_5_True_False_True_False): 0.02s I0421 03:58:35.970221 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_1_5_True_False_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_1_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_False_True): 0.01s I0421 03:58:35.980320 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_False): 0.01s I0421 03:58:35.990278 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_1_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_1_False_True_True_False): 0.02s I0421 03:58:36.009567 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_1_False_True_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_1_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_False_True): 0.01s I0421 03:58:36.019674 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_False): 0.01s I0421 03:58:36.029551 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_5_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_5_3_False_False_True_False): 0.01s I0421 03:58:36.039277 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_5_3_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_5_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_False_True): 0.01s I0421 03:58:36.048979 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_False): 0.01s I0421 03:58:36.058789 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_3_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_3_3_True_False_True_False): 0.01s I0421 03:58:36.069129 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_3_3_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_3_3_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_False_True): 0.01s I0421 03:58:36.079359 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_False): 0.01s I0421 03:58:36.089407 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_1_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_5_False_True_True_False): 0.01s I0421 03:58:36.099841 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_5_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_1_5_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_False_True): 0.01s I0421 03:58:36.109439 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_False): 0.01s I0421 03:58:36.119290 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_1_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_1_False_False_True_False): 0.01s I0421 03:58:36.134914 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_1_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_1_1_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_False_True): 0.01s I0421 03:58:36.150248 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_False): 0.02s I0421 03:58:36.169821 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_5_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_5_1_True_False_True_False): 0.01s I0421 03:58:36.182101 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_5_1_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_5_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_False_True): 0.01s I0421 03:58:36.193953 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_False): 0.01s I0421 03:58:36.209366 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_3_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_3_3_False_True_True_False): 0.02s I0421 03:58:36.229380 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_3_3_False_True_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_3_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_False_True): 0.01s I0421 03:58:36.241425 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_False): 0.01s I0421 03:58:36.256933 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_1_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_5_False_False_True_False): 0.01s I0421 03:58:36.269143 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_1_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_False_True): 0.01s I0421 03:58:36.284580 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_False): 0.01s I0421 03:58:36.300397 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_5_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_5_5_True_False_True_False): 0.03s I0421 03:58:36.327751 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_5_5_True_False_True_False): 0.03s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_5_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_False_True): 0.01s I0421 03:58:36.337241 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_False): 0.01s I0421 03:58:36.346616 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_5_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_1_False_True_True_False): 0.01s I0421 03:58:36.357083 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_1_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_5_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_False_True): 0.01s I0421 03:58:36.366744 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_False): 0.01s I0421 03:58:36.376394 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_3_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_3_3_False_False_True_False): 0.01s I0421 03:58:36.386082 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_3_3_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_3_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_False_True): 0.01s I0421 03:58:36.396057 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_False): 0.01s I0421 03:58:36.405824 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_1_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_1_3_True_False_True_False): 0.01s I0421 03:58:36.415518 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_1_3_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_1_3_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_False_True): 0.01s I0421 03:58:36.425223 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_False): 0.01s I0421 03:58:36.434840 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_5_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_5_False_True_True_False): 0.01s I0421 03:58:36.444870 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_5_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_5_5_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_False_True): 0.01s I0421 03:58:36.460808 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_False): 0.01s I0421 03:58:36.470996 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_5_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_1_False_False_True_False): 0.01s I0421 03:58:36.480623 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_1_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_5_1_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_False_True): 0.01s I0421 03:58:36.490138 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_False): 0.01s I0421 03:58:36.499976 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_3_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_3_1_True_False_True_False): 0.01s I0421 03:58:36.509424 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_3_1_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_3_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_False_True): 0.01s I0421 03:58:36.519308 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_False): 0.01s I0421 03:58:36.529133 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_1_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_1_3_False_True_True_False): 0.01s I0421 03:58:36.539529 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_1_3_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_1_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_False_True): 0.01s I0421 03:58:36.549474 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_False): 0.01s I0421 03:58:36.559016 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_5_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_5_False_False_True_False): 0.01s I0421 03:58:36.568527 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_5_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_False_True): 0.01s I0421 03:58:36.583298 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_False): 0.01s I0421 03:58:36.593011 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_3_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_3_5_True_False_True_False): 0.01s I0421 03:58:36.602789 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_3_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_3_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_False_True): 0.01s I0421 03:58:36.612331 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_False): 0.01s I0421 03:58:36.622087 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_3_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_1_False_True_True_False): 0.01s I0421 03:58:36.632781 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_1_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_3_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_False_True): 0.01s I0421 03:58:36.642790 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_False): 0.01s I0421 03:58:36.652460 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_1_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_1_3_False_False_True_False): 0.01s I0421 03:58:36.662211 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_1_3_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_1_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_False_True): 0.01s I0421 03:58:36.671814 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_False): 0.01s I0421 03:58:36.681514 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_5_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_5_3_True_False_True_False): 0.01s I0421 03:58:36.690903 281473501983584 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_5_3_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_5_3_True_False_True_False ====================================================================== FAIL: testMismatchedDimensions (__main__.MatMulInfixOperatorTest) MatMulInfixOperatorTest.testMismatchedDimensions ---------------------------------------------------------------------- tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped____MklMatMul_device_/job:localhost/replica:0/task:0/device:CPU:0}} lhs mismatch rhs shape: 3 vs. 2: [1,3] [2,2] 0 0 [Op:MatMul] name: During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py", line 226, in testMismatchedDimensions infix_matmul( AssertionError: "(Matrix size-incompatible: In\[0\]: .* In\[1\]|Dimensions must be equal)" does not match "{{function_node __wrapped____MklMatMul_device_/job:localhost/replica:0/task:0/device:CPU:0}} lhs mismatch rhs shape: 3 vs. 2: [1,3] [2,2] 0 0 [Op:MatMul] name: " ---------------------------------------------------------------------- Ran 195 tests in 11.093s FAILED (failures=1, skipped=48) ================================================================================ ==================== Test output for //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu (shard 8 of 20): 2023-04-21 03:58:38.600586: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:266: ComplexWarning: Casting complex values to real discards the imaginary part a_np.imag = np.random.normal(-5, 5, /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:270: ComplexWarning: Casting complex values to real discards the imaginary part b_np.imag = np.random.normal(-5, 5, Running tests under Python 3.8.10: /usr/local/bin/python3 [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_False_True WARNING:tensorflow:From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:153: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.config.list_physical_devices('GPU')` instead. W0421 03:58:40.616845 281473436578656 deprecation.py:364] From /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:153: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.config.list_physical_devices('GPU')` instead. INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_False_True): 0.01s I0421 03:58:40.623025 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_False_True): 0.01s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_1_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_False): 0.0s I0421 03:58:40.623953 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_False_True_False): 0.0s I0421 03:58:40.624595 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_3_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_False_True): 0.0s I0421 03:58:40.625196 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_1_5_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_False): 0.0s I0421 03:58:40.625795 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_5_True_False_True_False): 0.0s I0421 03:58:40.626430 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_5_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_1_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_False_True): 0.0s I0421 03:58:40.627008 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_3_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_False): 0.0s I0421 03:58:40.627585 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_3_5_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_False_True_True_False): 0.0s I0421 03:58:40.628163 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_False_True): 0.0s I0421 03:58:40.628739 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_1_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_False): 0.0s I0421 03:58:40.629313 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_3_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_False_True_False): 0.0s I0421 03:58:40.629887 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_bfloat16_5_5_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_False_True 2023-04-21 03:58:40.668746: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:375] MLIR V1 optimization pass is not enabled INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_False_True): 0.18s I0421 03:58:40.806663 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_False_True): 0.18s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_False): 0.13s I0421 03:58:40.937042 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_False): 0.13s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_1_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_3_True_False_True_False): 0.15s I0421 03:58:41.087331 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_3_True_False_True_False): 0.15s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_3_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_False_True): 0.15s I0421 03:58:41.235925 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_False_True): 0.15s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_1_5_3_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_False): 0.11s I0421 03:58:41.341963 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_False): 0.11s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_1_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_False_True_True_False): 0.17s I0421 03:58:41.513017 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_False_True_True_False): 0.17s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_1_5_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_False_True): 0.21s I0421 03:58:41.719062 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_False_True): 0.21s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_3_3_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_False): 0.22s I0421 03:58:41.941777 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_False): 0.22s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_3_5_3_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_False_True_False): 0.13s I0421 03:58:42.076992 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_False_True_False): 0.13s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_1_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_False_True): 0.19s I0421 03:58:42.266679 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_False_True): 0.19s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_1_5_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_False): 0.31s I0421 03:58:42.580600 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_False): 0.31s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_3_3_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_1_True_False_True_False): 0.34s I0421 03:58:42.921269 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_1_True_False_True_False): 0.34s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex128_5_5_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_False_True): 0.22s I0421 03:58:43.137740 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_False_True): 0.22s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_False): 0.11s I0421 03:58:43.248254 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_False): 0.11s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_1_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_False_True_True_False): 0.15s I0421 03:58:43.395048 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_False_True_True_False): 0.15s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_3_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_False_True): 0.14s I0421 03:58:43.531733 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_False_True): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_1_5_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_False): 0.12s I0421 03:58:43.655415 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_False): 0.12s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_False_True_False): 0.16s I0421 03:58:43.815710 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_False_True_False): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_1_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_False_True): 0.17s I0421 03:58:43.983019 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_False_True): 0.17s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_3_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_False): 0.19s I0421 03:58:44.177536 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_False): 0.19s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_5_True_False_True_False): 0.28s I0421 03:58:44.456401 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_5_True_False_True_False): 0.28s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_3_5_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_False_True): 0.25s I0421 03:58:44.710346 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_False_True): 0.25s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_1_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_False): 0.29s I0421 03:58:45.000909 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_False): 0.29s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_3_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_False_True_True_False): 0.28s I0421 03:58:45.278402 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_False_True_True_False): 0.28s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_False_True): 0.36s I0421 03:58:45.635176 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_False_True): 0.36s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_complex64_5_5_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_False): 0.0s I0421 03:58:45.636062 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_1_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_False_True_False): 0.0s I0421 03:58:45.643548 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_3_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_False_True): 0.0s I0421 03:58:45.644135 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_False): 0.0s I0421 03:58:45.644663 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_1_5_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_3_True_False_True_False): 0.0s I0421 03:58:45.645184 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_3_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_1_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_False_True): 0.0s I0421 03:58:45.645699 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_3_3_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_False): 0.0s I0421 03:58:45.646249 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_1_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_False_True_True_False): 0.0s I0421 03:58:45.646785 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_3_5_5_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_False_True): 0.0s I0421 03:58:45.647297 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_1_3_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_False): 0.0s I0421 03:58:45.647823 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_3_3_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_False_True_False): 0.0s I0421 03:58:45.648334 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_1_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_False_True): 0.0s I0421 03:58:45.648842 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_float16_5_5_5_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_False): 0.14s I0421 03:58:45.792736 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_1_3_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_1_True_False_True_False): 0.18s I0421 03:58:45.975803 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_1_True_False_True_False): 0.18s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_3_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_False_True): 0.13s I0421 03:58:46.110189 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_False_True): 0.13s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_False): 0.18s I0421 03:58:46.295808 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_False): 0.18s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_1_5_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_False_True_True_False): 0.13s I0421 03:58:46.430729 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_False_True_True_False): 0.13s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_1_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_False_True): 0.14s I0421 03:58:46.568551 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_False_True): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_3_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_False): 0.17s I0421 03:58:46.735256 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_False): 0.17s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_False_True_False): 0.21s I0421 03:58:46.941740 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_False_True_False): 0.21s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_3_5_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_False_True): 0.14s I0421 03:58:47.083893 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_False_True): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_1_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_False): 0.16s I0421 03:58:47.244240 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_False): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_5_True_False_True_False): 0.22s I0421 03:58:47.463937 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_5_True_False_True_False): 0.22s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_3_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_False_True): 0.28s I0421 03:58:47.740204 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_False_True): 0.28s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float32_5_5_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_False): 0.09s I0421 03:58:47.829183 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_False): 0.09s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_1_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_False_True_True_False): 0.09s I0421 03:58:47.915185 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_False_True_True_False): 0.09s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_False_True): 0.17s I0421 03:58:48.081962 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_False_True): 0.17s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_3_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_False): 0.14s I0421 03:58:48.218454 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_1_5_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_False_True_False): 0.1s I0421 03:58:48.318806 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_False_True_False): 0.1s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_1_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_False_True): 0.12s I0421 03:58:48.435195 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_False_True): 0.12s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_False): 0.13s I0421 03:58:48.566309 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_False): 0.13s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_3_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_3_True_False_True_False): 0.15s I0421 03:58:48.716819 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_3_True_False_True_False): 0.15s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_3_5_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_False_True): 0.13s I0421 03:58:48.843404 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_False_True): 0.13s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_1_3_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_False): 0.14s I0421 03:58:48.986604 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_False): 0.14s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_1_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_False_True_True_False): 0.16s I0421 03:58:49.149094 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_False_True_True_False): 0.16s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_3_5_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_False_True): 0.17s I0421 03:58:49.320651 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_False_True): 0.17s [ OK ] MatMulGradientTest.test_MatMulGradientTest_True_float64_5_5_3_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_False): 0.0s I0421 03:58:49.321565 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_1_3_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_False_True_False): 0.0s I0421 03:58:49.322120 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_1_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_False_True): 0.0s I0421 03:58:49.322647 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_3_5_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_False): 0.0s I0421 03:58:49.323161 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_1_5_3_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_1_True_False_True_False): 0.0s I0421 03:58:49.323672 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_1_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_1_1_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_False_True): 0.0s I0421 03:58:49.324186 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_1_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_False): 0.0s I0421 03:58:49.324699 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_3_5_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_False_True_True_False): 0.0s I0421 03:58:49.325222 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_3_5_3_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_False_True): 0.0s I0421 03:58:49.325732 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_1_1_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_False): 0.0s I0421 03:58:49.326269 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_1_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_False_True_False): 0.0s I0421 03:58:49.326802 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_3_5_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_False_True): 0.0s I0421 03:58:49.327311 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int32_5_5_3_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_False): 0.0s I0421 03:58:49.327823 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_1_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_5_True_False_True_False): 0.0s I0421 03:58:49.328333 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_5_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_1_5_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_False_True): 0.0s I0421 03:58:49.328841 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_3_5_False_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_False): 0.0s I0421 03:58:49.329349 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_1_5_3_False_True_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_False_True_True_False): 0.0s I0421 03:58:49.329869 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_False_True_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_1_False_True_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_False_True): 0.0s I0421 03:58:49.330377 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_1_5_True_False_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_False): 0.0s I0421 03:58:49.330890 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_3_5_False_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_False_True_False): 0.0s I0421 03:58:49.331403 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_3_5_3_False_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_False_True): 0.0s I0421 03:58:49.331914 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_1_False_True_False_True [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_False): 0.0s I0421 03:58:49.332420 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_1_5_True_False_False_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_3_True_False_True_False): 0.0s I0421 03:58:49.332926 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_3_True_False_True_False): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_3_3_True_False_True_False [ RUN ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_False_True): 0.0s I0421 03:58:49.333436 281473436578656 test_util.py:2462] time(__main__.MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_False_True): 0.0s [ SKIPPED ] MatMulGradientTest.test_MatMulGradientTest_True_int64_5_5_3_False_False_False_True [ RUN ] MatMulInfixOperatorTest.testMismatchedDimensions INFO:tensorflow:time(__main__.MatMulInfixOperatorTest.testMismatchedDimensions): 0.03s I0421 03:58:49.363636 281473436578656 test_util.py:2462] time(__main__.MatMulInfixOperatorTest.testMismatchedDimensions): 0.03s [ FAILED ] MatMulInfixOperatorTest.testMismatchedDimensions [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_False_True_True_False 2023-04-21 03:58:49.375971: E ./tensorflow/core/graph/mkl_graph_util.h:182] oneDNN BFloat16 support are only on platforms with AVX512. Falling back to default implementation if present. /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py:103: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. np_val = np.matrix(a_np_) * np.matrix(b_np_) /usr/local/lib/python3.8/dist-packages/numpy/matrixlib/defmatrix.py:69: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray. return matrix(data, dtype=dtype, copy=False) INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_False_True_True_False): 0.01s I0421 03:58:49.378945 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_1_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_False_True): 0.01s I0421 03:58:49.387337 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_3_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_False): 0.01s I0421 03:58:49.396517 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_False_True_False): 0.01s I0421 03:58:49.405055 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_1_5_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_False_True): 0.01s I0421 03:58:49.413335 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_1_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_False): 0.01s I0421 03:58:49.421513 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_3_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_3_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_3_5_True_False_True_False): 0.01s I0421 03:58:49.429803 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_3_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_3_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_False_True): 0.01s I0421 03:58:49.437862 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_3_5_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_False): 0.01s I0421 03:58:49.445735 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_1_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_False_True_True_False): 0.01s I0421 03:58:49.453871 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_False_True): 0.01s I0421 03:58:49.461778 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_3_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_False): 0.01s I0421 03:58:49.475055 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_bfloat16_5_5_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_False_True_False): 0.01s I0421 03:58:49.490276 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_1_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_False_True): 0.01s I0421 03:58:49.498611 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_3_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_False): 0.01s I0421 03:58:49.514395 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_3_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_1_5_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_5_3_True_False_True_False): 0.02s I0421 03:58:49.531498 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_1_5_3_True_False_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex128_1_5_3_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_False_True): 0.01s I0421 03:58:49.539860 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_1_3_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_False): 0.01s I0421 03:58:49.548081 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_3_1_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_3_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_5_False_True_True_False): 0.01s I0421 03:58:49.562453 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_3_5_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_3_5_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_False_True): 0.01s I0421 03:58:49.578381 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_3_5_3_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_False): 0.01s I0421 03:58:49.587799 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_1_3_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_False_True_False): 0.01s I0421 03:58:49.602707 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_3_1_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_False_True): 0.01s I0421 03:58:49.610762 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_3_5_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_False): 0.01s I0421 03:58:49.625701 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex128_5_5_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_1_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_1_1_True_False_True_False): 0.02s I0421 03:58:49.642043 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_1_1_True_False_True_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_1_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_False_True): 0.01s I0421 03:58:49.650046 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_3_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_False): 0.01s I0421 03:58:49.657929 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_3_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_1_5_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_5_3_False_True_True_False): 0.01s I0421 03:58:49.673342 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_1_5_3_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_1_5_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_False_True): 0.02s I0421 03:58:49.689444 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_False_True): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_1_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_False): 0.01s I0421 03:58:49.697633 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_3_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_False_True_False): 0.01s I0421 03:58:49.712789 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_3_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_False_True): 0.01s I0421 03:58:49.723331 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_3_5_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_False): 0.01s I0421 03:58:49.738413 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_1_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_1_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_1_5_True_False_True_False): 0.01s I0421 03:58:49.753977 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_1_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_1_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_False_True): 0.01s I0421 03:58:49.762371 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_3_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_False): 0.01s I0421 03:58:49.771040 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_complex64_5_5_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_1_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_1_False_True_True_False): 0.01s I0421 03:58:49.782819 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_1_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_1_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_False_True): 0.01s I0421 03:58:49.790889 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_1_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_False): 0.01s I0421 03:58:49.798732 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_3_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_1_5_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_1_5_3_False_False_True_False): 0.01s I0421 03:58:49.807149 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_1_5_3_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_1_5_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_False_True): 0.01s I0421 03:58:49.814970 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_1_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_False): 0.01s I0421 03:58:49.823177 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_1_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_3_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_3_3_True_False_True_False): 0.01s I0421 03:58:49.831266 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_3_3_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_3_3_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_False_True): 0.01s I0421 03:58:49.839397 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_3_5_3_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_False): 0.01s I0421 03:58:49.847398 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_1_1_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_1_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_5_False_True_True_False): 0.01s I0421 03:58:49.856106 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_1_5_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_1_5_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_False_True): 0.01s I0421 03:58:49.864450 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_3_3_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_False): 0.01s I0421 03:58:49.872819 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float16_5_5_3_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_1_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_1_False_False_True_False): 0.01s I0421 03:58:49.886777 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_1_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_1_1_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_False_True): 0.02s I0421 03:58:49.904437 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_False_True): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_1_5_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_False): 0.02s I0421 03:58:49.922556 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_3_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_1_5_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_1_5_1_True_False_True_False): 0.01s I0421 03:58:49.936534 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_1_5_1_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_1_5_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_False_True): 0.01s I0421 03:58:49.946922 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_1_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_False): 0.02s I0421 03:58:49.964270 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_False): 0.02s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_1_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_3_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_3_3_False_True_True_False): 0.01s I0421 03:58:49.978717 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_3_3_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_3_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_False_True): 0.01s I0421 03:58:49.988995 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_3_5_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_False): 0.01s I0421 03:58:50.002910 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_1_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_1_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_5_False_False_True_False): 0.01s I0421 03:58:50.013487 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_1_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_1_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_False_True): 0.01s I0421 03:58:50.027106 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_3_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_False): 0.01s I0421 03:58:50.041309 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_5_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float32_5_5_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float32_5_5_5_True_False_True_False): 0.01s I0421 03:58:50.055273 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float32_5_5_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float32_5_5_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_False_True): 0.01s I0421 03:58:50.063349 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_1_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_False): 0.01s I0421 03:58:50.071340 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_3_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_5_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_1_False_True_True_False): 0.01s I0421 03:58:50.079936 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_1_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_5_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_False_True): 0.01s I0421 03:58:50.087945 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_1_5_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_False): 0.01s I0421 03:58:50.095842 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_1_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_3_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_3_3_False_False_True_False): 0.01s I0421 03:58:50.103765 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_3_3_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_3_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_False_True): 0.01s I0421 03:58:50.111440 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_5_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_False): 0.01s I0421 03:58:50.119558 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_3_5_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_1_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_1_3_True_False_True_False): 0.01s I0421 03:58:50.127367 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_1_3_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_1_3_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_False_True): 0.01s I0421 03:58:50.137974 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_3_3_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_False): 0.01s I0421 03:58:50.146060 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_5_1_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_float64_5_5_5_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_5_False_True_True_False): 0.01s I0421 03:58:50.154835 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_float64_5_5_5_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_float64_5_5_5_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_False_True): 0.01s I0421 03:58:50.165547 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_1_3_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_False): 0.01s I0421 03:58:50.173680 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_3_3_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_5_1_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_1_False_False_True_False): 0.01s I0421 03:58:50.181772 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_1_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_5_1_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_False_True): 0.01s I0421 03:58:50.189842 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_1_5_5_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_False): 0.03s I0421 03:58:50.217426 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_False): 0.03s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_1_3_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_3_1_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_3_1_True_False_True_False): 0.01s I0421 03:58:50.230903 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_3_1_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_3_1_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_False_True): 0.01s I0421 03:58:50.238752 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_5_1_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_False): 0.01s I0421 03:58:50.246703 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_3_5_5_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_1_3_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_1_3_False_True_True_False): 0.01s I0421 03:58:50.255464 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_1_3_False_True_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_1_3_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_False_True): 0.01s I0421 03:58:50.263582 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_3_1_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_False): 0.01s I0421 03:58:50.271600 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_5_1_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int32_5_5_5_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_5_False_False_True_False): 0.01s I0421 03:58:50.281028 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int32_5_5_5_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int32_5_5_5_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_False_True): 0.01s I0421 03:58:50.291313 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_1_3_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_False): 0.01s I0421 03:58:50.299306 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_3_1_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_3_5_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_3_5_True_False_True_False): 0.01s I0421 03:58:50.307175 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_3_5_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_3_5_True_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_False_True): 0.01s I0421 03:58:50.316683 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_1_5_5_False_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_False): 0.01s I0421 03:58:50.324650 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_1_3_False_True_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_3_1_False_True_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_1_False_True_True_False): 0.03s I0421 03:58:50.350847 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_1_False_True_True_False): 0.03s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_3_1_False_True_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_False_True): 0.01s I0421 03:58:50.358614 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_3_5_True_False_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_False): 0.01s I0421 03:58:50.366156 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_3_5_5_False_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_1_3_False_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_1_3_False_False_True_False): 0.01s I0421 03:58:50.373845 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_1_3_False_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_1_3_False_False_True_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_False_True INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_False_True): 0.01s I0421 03:58:50.382361 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_False_True): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_3_1_False_True_False_True [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_False): 0.01s I0421 03:58:50.390290 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_3_5_True_False_False_False [ RUN ] MatMulTest.test_MatMulTest_True_int64_5_5_3_True_False_True_False INFO:tensorflow:time(__main__.MatMulTest.test_MatMulTest_True_int64_5_5_3_True_False_True_False): 0.01s I0421 03:58:50.397911 281473436578656 test_util.py:2462] time(__main__.MatMulTest.test_MatMulTest_True_int64_5_5_3_True_False_True_False): 0.01s [ OK ] MatMulTest.test_MatMulTest_True_int64_5_5_3_True_False_True_False ====================================================================== FAIL: testMismatchedDimensions (__main__.MatMulInfixOperatorTest) MatMulInfixOperatorTest.testMismatchedDimensions ---------------------------------------------------------------------- tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped____MklMatMul_device_/job:localhost/replica:0/task:0/device:CPU:0}} lhs mismatch rhs shape: 3 vs. 2: [1,3] [2,2] 0 0 [Op:MatMul] name: During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/matmul_op_test.py", line 226, in testMismatchedDimensions infix_matmul( AssertionError: "(Matrix size-incompatible: In\[0\]: .* In\[1\]|Dimensions must be equal)" does not match "{{function_node __wrapped____MklMatMul_device_/job:localhost/replica:0/task:0/device:CPU:0}} lhs mismatch rhs shape: 3 vs. 2: [1,3] [2,2] 0 0 [Op:MatMul] name: " ---------------------------------------------------------------------- Ran 195 tests in 9.783s FAILED (failures=1, skipped=48) ================================================================================ ==================== Test output for //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu (shard 2 of 20): 2023-04-21 03:59:49.988871: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. Running tests under Python 3.8.10: /usr/local/bin/python3 [ RUN ] TensordotTest.test_invalid_shape INFO:tensorflow:Running test_invalid_shape in GRAPH mode. I0421 03:59:56.143910 281473816032096 test_util.py:1492] Running test_invalid_shape in GRAPH mode. WARNING:tensorflow:From /usr/lib/python3.8/contextlib.py:83: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `self.session()` or `self.cached_session()` instead. W0421 03:59:56.144459 281473816032096 deprecation.py:364] From /usr/lib/python3.8/contextlib.py:83: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `self.session()` or `self.cached_session()` instead. 2023-04-21 03:59:56.374325: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:375] MLIR V1 optimization pass is not enabled 2023-04-21 03:59:56.433763: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] ERROR:tensorflow:Graph execution error: Detected at node 'Tensordot_1/MatMul' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) Node: 'Tensordot_1/MatMul' lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] Original stack trace for 'Tensordot_1/MatMul': File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main true_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 312, in run _run_main(main, args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1496, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 5188, in tensordot ab_matmul = matmul(a_reshape, b_reshape) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 3794, in matmul return gen_math_ops.mat_mul( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/gen_math_ops.py", line 6035, in mat_mul _, _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 795, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 3383, in _create_op_internal ret = Operation.from_node_def( E0421 03:59:56.982346 281473816032096 test_util.py:2065] Graph execution error: Detected at node 'Tensordot_1/MatMul' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) Node: 'Tensordot_1/MatMul' lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] Original stack trace for 'Tensordot_1/MatMul': File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main true_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 312, in run _run_main(main, args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1496, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 5188, in tensordot ab_matmul = matmul(a_reshape, b_reshape) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 3794, in matmul return gen_math_ops.mat_mul( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/gen_math_ops.py", line 6035, in mat_mul _, _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 795, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 3383, in _create_op_internal ret = Operation.from_node_def( INFO:tensorflow:Searching within error strings: 'Matrix size-incompatible: In\[0\]: \[2,2\], In\[1\]: \[3,2\]' within 'Graph execution error: Detected at node 'Tensordot_1/MatMul' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) Node: 'Tensordot_1/MatMul' lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] Caused by: Tensordot_1/MatMul' I0421 03:59:56.982693 281473816032096 test_util.py:3533] Searching within error strings: 'Matrix size-incompatible: In\[0\]: \[2,2\], In\[1\]: \[3,2\]' within 'Graph execution error: Detected at node 'Tensordot_1/MatMul' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) Node: 'Tensordot_1/MatMul' lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] Caused by: Tensordot_1/MatMul' INFO:tensorflow:time(__main__.TensordotTest.test_invalid_shape): 0.84s I0421 03:59:56.985076 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_invalid_shape): 0.84s [ FAILED ] TensordotTest.test_invalid_shape [ RUN ] TensordotTest.test_test_tensordot_complex128_2_1_0_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 03:59:56.992100 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_2_1_0_True): 0.0s I0421 03:59:56.993147 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 03:59:56.993736 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_2_1_0_True): 0.0s I0421 03:59:56.994019 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex128_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_complex128_2_5_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 03:59:56.994591 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py:147: DeprecationWarning: This function is deprecated. Please call randint(1, 5 + 1) instead a_shape = np.random.random_integers(1, _MAXDIM, rank_a_) /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py:148: DeprecationWarning: This function is deprecated. Please call randint(1, 5 + 1) instead b_shape = np.random.random_integers(1, _MAXDIM, rank_b_) /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py:149: DeprecationWarning: This function is deprecated. Please call randint(1, 5 + 1) instead shared_shape = np.random.random_integers(1, _MAXDIM, num_dims_) INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_2_5_2_True): 0.44s I0421 03:59:57.437970 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_2_5_2_True): 0.44s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 03:59:57.440701 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_2_5_2_True): 0.0s I0421 03:59:57.441083 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex128_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_complex128_4_4_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 03:59:57.441704 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_4_4_4_True): 1.8s I0421 03:59:59.237599 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_4_4_4_True): 1.8s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 03:59:59.247183 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_4_4_4_True): 0.0s I0421 03:59:59.247641 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex128_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_complex128_5_2_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 03:59:59.248314 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_5_2_2_True): 0.39s I0421 03:59:59.637867 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_5_2_2_True): 0.39s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 03:59:59.640964 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_5_2_2_True): 0.0s I0421 03:59:59.641359 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex128_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_complex128_5_5_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 03:59:59.641977 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_5_5_4_True): 1.99s I0421 04:00:01.627966 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_5_5_4_True): 1.99s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:01.636388 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_5_5_4_True): 0.0s I0421 04:00:01.636759 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex128_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_complex64_2_1_0_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:01.637357 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_2_1_0_True): 0.0s I0421 04:00:01.638172 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:01.638657 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_2_1_0_True): 0.0s I0421 04:00:01.638903 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex64_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_complex64_2_5_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:01.639398 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_2_5_2_True): 0.44s I0421 04:00:02.082705 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_2_5_2_True): 0.44s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:02.086001 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_2_5_2_True): 0.0s I0421 04:00:02.086476 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex64_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_complex64_4_4_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:02.087092 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_4_4_4_True): 1.89s I0421 04:00:03.981787 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_4_4_4_True): 1.89s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:03.996218 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_4_4_4_True): 0.0s I0421 04:00:03.996815 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex64_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_complex64_5_2_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:03.997523 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_5_2_2_True): 0.4s I0421 04:00:04.402496 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_5_2_2_True): 0.4s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:04.407070 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_5_2_2_True): 0.0s I0421 04:00:04.407600 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex64_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_complex64_5_5_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:04.408279 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_5_5_4_True): 1.92s I0421 04:00:06.328206 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_5_5_4_True): 1.92s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:06.337832 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_5_5_4_True): 0.0s I0421 04:00:06.338362 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex64_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_float16_2_1_0_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:06.339000 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_2_1_0_True): 0.0s I0421 04:00:06.339907 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:06.340406 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_2_1_0_True): 0.0s I0421 04:00:06.340658 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float16_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_float16_2_5_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:06.341176 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_2_5_2_True): 0.5s I0421 04:00:06.837704 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_2_5_2_True): 0.5s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:06.841190 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_2_5_2_True): 0.0s I0421 04:00:06.841675 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float16_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_float16_4_4_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:06.842362 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_4_4_4_True): 1.74s I0421 04:00:08.582553 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_4_4_4_True): 1.74s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:08.592564 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_4_4_4_True): 0.0s I0421 04:00:08.593092 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float16_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_float16_5_2_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:08.593754 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_5_2_2_True): 0.39s I0421 04:00:08.979691 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_5_2_2_True): 0.39s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:08.983157 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_5_2_2_True): 0.0s I0421 04:00:08.983689 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float16_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_float16_5_5_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:08.984370 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_5_5_4_True): 1.76s I0421 04:00:10.748370 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_5_5_4_True): 1.76s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:10.758202 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_5_5_4_True): 0.0s I0421 04:00:10.758770 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float16_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_float32_2_1_0_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:10.759466 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_2_1_0_True): 0.0s I0421 04:00:10.760561 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:10.761155 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_2_1_0_True): 0.0s I0421 04:00:10.761433 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float32_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_float32_2_5_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:10.762039 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_2_5_2_True): 0.51s I0421 04:00:11.267688 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_2_5_2_True): 0.51s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:11.271042 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_2_5_2_True): 0.0s I0421 04:00:11.271517 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float32_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_float32_4_4_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:11.272192 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_4_4_4_True): 2.33s I0421 04:00:13.597211 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_4_4_4_True): 2.33s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:13.610822 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_4_4_4_True): 0.0s I0421 04:00:13.611261 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float32_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_float32_5_2_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:13.611893 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_5_2_2_True): 0.65s I0421 04:00:14.265904 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_5_2_2_True): 0.65s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:14.270179 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_5_2_2_True): 0.0s I0421 04:00:14.270558 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float32_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_float32_5_5_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:14.271181 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_5_5_4_True): 2.75s I0421 04:00:17.021376 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_5_5_4_True): 2.75s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:17.033155 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_5_5_4_True): 0.0s I0421 04:00:17.033713 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float32_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_float64_2_1_0_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:17.034399 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_2_1_0_True): 0.0s I0421 04:00:17.035443 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:17.036023 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_2_1_0_True): 0.0s I0421 04:00:17.036805 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float64_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_float64_2_5_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:17.037422 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_2_5_2_True): 0.69s I0421 04:00:17.731070 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_2_5_2_True): 0.69s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:17.734572 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_2_5_2_True): 0.0s I0421 04:00:17.734984 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float64_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_float64_4_4_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:17.735623 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_4_4_4_True): 2.35s I0421 04:00:20.084363 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_4_4_4_True): 2.35s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:20.095191 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_4_4_4_True): 0.0s I0421 04:00:20.095692 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float64_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_float64_5_2_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:20.096385 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_5_2_2_True): 0.45s I0421 04:00:20.545682 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_5_2_2_True): 0.45s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:20.549141 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_5_2_2_True): 0.0s I0421 04:00:20.549506 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float64_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_float64_5_5_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:20.550105 281473816032096 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_5_5_4_True): 3.06s I0421 04:00:23.612177 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_5_5_4_True): 3.06s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:23.622725 281473816032096 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_5_5_4_True): 0.0s I0421 04:00:23.623153 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float64_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex128_2_1_0_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:23.623783 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_1_0_True): 0.0s I0421 04:00:23.624667 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:23.625211 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_1_0_True): 0.0s I0421 04:00:23.625474 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex128_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex128_2_5_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:23.626046 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_5_2_True): 0.23s I0421 04:00:23.857354 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_5_2_True): 0.23s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:23.859419 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_5_2_True): 0.0s I0421 04:00:23.859726 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex128_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex128_4_4_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:23.860315 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_4_4_4_True): 2.94s I0421 04:00:26.802288 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_4_4_4_True): 2.94s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:26.804494 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_4_4_4_True): 0.0s I0421 04:00:26.804834 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex128_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex128_5_2_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:26.805430 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_2_2_True): 0.16s I0421 04:00:26.966998 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_2_2_True): 0.16s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:26.968725 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_2_2_True): 0.0s I0421 04:00:26.969076 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex128_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex128_5_5_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:26.969683 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_5_4_True): 2.2s I0421 04:00:29.167622 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_5_4_True): 2.2s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:29.171394 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_5_4_True): 0.0s I0421 04:00:29.171864 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex128_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex64_2_1_0_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:29.172548 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_1_0_True): 0.0s I0421 04:00:29.173569 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:29.174123 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_1_0_True): 0.0s I0421 04:00:29.174387 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex64_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex64_2_5_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:29.174959 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_5_2_True): 1.49s I0421 04:00:30.669233 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_5_2_True): 1.49s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:30.672036 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_5_2_True): 0.0s I0421 04:00:30.672482 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex64_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex64_4_4_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:30.673120 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_4_4_4_True): 3.24s I0421 04:00:33.909865 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_4_4_4_True): 3.24s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:33.912477 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_4_4_4_True): 0.0s I0421 04:00:33.912811 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex64_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex64_5_2_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:33.913434 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_2_2_True): 0.64s I0421 04:00:34.551964 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_2_2_True): 0.64s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:34.553951 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_2_2_True): 0.0s I0421 04:00:34.554268 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex64_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex64_5_5_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:34.554893 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_5_4_True): 2.94s I0421 04:00:37.497682 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_5_4_True): 2.94s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:37.500354 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_5_4_True): 0.0s I0421 04:00:37.500663 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex64_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float16_2_1_0_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:37.501276 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_1_0_True): 0.0s I0421 04:00:37.502116 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:37.502593 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_1_0_True): 0.0s I0421 04:00:37.502841 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float16_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float16_2_5_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:37.503337 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_5_2_True): 0.35s I0421 04:00:37.855993 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_5_2_True): 0.35s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:37.857991 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_5_2_True): 0.0s I0421 04:00:37.858368 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float16_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float16_4_4_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:37.859011 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_4_4_4_True): 0.4s I0421 04:00:38.257705 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_4_4_4_True): 0.4s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:38.260221 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_4_4_4_True): 0.0s I0421 04:00:38.260537 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float16_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float16_5_2_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:38.261165 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_2_2_True): 0.16s I0421 04:00:38.424011 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_2_2_True): 0.16s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:38.425902 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_2_2_True): 0.0s I0421 04:00:38.426246 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float16_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float16_5_5_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:38.426895 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_5_4_True): 0.6s I0421 04:00:39.029322 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_5_4_True): 0.6s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:39.032267 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_5_4_True): 0.0s I0421 04:00:39.032699 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float16_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float32_2_1_0_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:39.033343 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_1_0_True): 0.0s I0421 04:00:39.034314 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:39.034838 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_1_0_True): 0.0s I0421 04:00:39.035094 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float32_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float32_2_5_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:39.035666 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_5_2_True): 0.2s I0421 04:00:39.231817 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_5_2_True): 0.2s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:39.234165 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_5_2_True): 0.0s I0421 04:00:39.234518 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float32_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float32_4_4_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:39.235152 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_4_4_4_True): 1.69s I0421 04:00:40.927612 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_4_4_4_True): 1.69s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:40.930492 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_4_4_4_True): 0.0s I0421 04:00:40.930892 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float32_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float32_5_2_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:40.931526 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_2_2_True): 0.64s I0421 04:00:41.567697 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_2_2_True): 0.64s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:41.569995 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_2_2_True): 0.0s I0421 04:00:41.570401 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float32_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float32_5_5_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:41.571035 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_5_4_True): 1.38s I0421 04:00:42.947600 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_5_4_True): 1.38s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:42.950325 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_5_4_True): 0.0s I0421 04:00:42.950656 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float32_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float64_2_1_0_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:42.951272 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_1_0_True): 0.0s I0421 04:00:42.952137 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:42.952630 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_1_0_True): 0.0s I0421 04:00:42.952885 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float64_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float64_2_5_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:42.953391 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_5_2_True): 0.31s I0421 04:00:43.261018 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_5_2_True): 0.31s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:43.263024 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_5_2_True): 0.0s I0421 04:00:43.263411 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float64_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float64_4_4_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:43.264054 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_4_4_4_True): 0.84s I0421 04:00:44.103429 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_4_4_4_True): 0.84s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:44.106476 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_4_4_4_True): 0.0s I0421 04:00:44.106999 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float64_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float64_5_2_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:44.107692 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_2_2_True): 0.2s I0421 04:00:44.305642 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_2_2_True): 0.2s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:44.307907 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_2_2_True): 0.0s I0421 04:00:44.308351 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float64_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:00:44.309010 281473816032096 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_4_True): 0.81s I0421 04:00:45.115038 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_4_True): 0.81s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:00:45.117917 281473816032096 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_4_True): 0.0s I0421 04:00:45.118380 281473816032096 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_4_True ====================================================================== FAIL: test_invalid_shape (__main__.TensordotTest) TensordotTest.test_invalid_shape ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1379, in _do_call return fn(*args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1362, in _run_fn return self._call_tf_sessionrun(options, feed_dict, fetch_list, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1455, in _call_tf_sessionrun return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict, tensorflow.python.framework.errors_impl.InvalidArgumentError: lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 3538, in assertRaisesWithPredicateMatch yield File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 60, in test_invalid_shape _ = sess.run( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 2059, in run return super().run(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 969, in run result = self._run(None, fetches, feed_dict, options_ptr, tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error: Detected at node 'Tensordot_1/MatMul' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) Node: 'Tensordot_1/MatMul' lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] Original stack trace for 'Tensordot_1/MatMul': File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main true_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 312, in run _run_main(main, args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1496, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 5188, in tensordot ab_matmul = matmul(a_reshape, b_reshape) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 3794, in matmul return gen_math_ops.mat_mul( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/gen_math_ops.py", line 6035, in mat_mul _, _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 795, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 3383, in _create_op_internal ret = Operation.from_node_def( During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1496, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 60, in test_invalid_shape _ = sess.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 3542, in assertRaisesWithPredicateMatch raise AssertionError("Exception of type %s: %s" % AssertionError: Exception of type : Graph execution error: Detected at node 'Tensordot_1/MatMul' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) Node: 'Tensordot_1/MatMul' lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] Original stack trace for 'Tensordot_1/MatMul': File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main true_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 312, in run _run_main(main, args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1496, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 5188, in tensordot ab_matmul = matmul(a_reshape, b_reshape) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 3794, in matmul return gen_math_ops.mat_mul( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/gen_math_ops.py", line 6035, in mat_mul _, _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 795, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 3383, in _create_op_internal ret = Operation.from_node_def( ---------------------------------------------------------------------- Ran 51 tests in 48.976s FAILED (failures=1, skipped=50) ================================================================================ ==================== Test output for //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu (shard 2 of 20): 2023-04-21 04:00:47.176449: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. Running tests under Python 3.8.10: /usr/local/bin/python3 [ RUN ] TensordotTest.test_invalid_shape INFO:tensorflow:Running test_invalid_shape in GRAPH mode. I0421 04:00:48.887305 281473637577568 test_util.py:1492] Running test_invalid_shape in GRAPH mode. WARNING:tensorflow:From /usr/lib/python3.8/contextlib.py:83: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `self.session()` or `self.cached_session()` instead. W0421 04:00:48.887752 281473637577568 deprecation.py:364] From /usr/lib/python3.8/contextlib.py:83: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `self.session()` or `self.cached_session()` instead. 2023-04-21 04:00:49.007185: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:375] MLIR V1 optimization pass is not enabled 2023-04-21 04:00:49.062911: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] ERROR:tensorflow:Graph execution error: Detected at node 'Tensordot_1/MatMul' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) Node: 'Tensordot_1/MatMul' lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] Original stack trace for 'Tensordot_1/MatMul': File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main true_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 312, in run _run_main(main, args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1496, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 5188, in tensordot ab_matmul = matmul(a_reshape, b_reshape) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 3794, in matmul return gen_math_ops.mat_mul( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/gen_math_ops.py", line 6035, in mat_mul _, _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 795, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 3383, in _create_op_internal ret = Operation.from_node_def( E0421 04:00:49.518584 281473637577568 test_util.py:2065] Graph execution error: Detected at node 'Tensordot_1/MatMul' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) Node: 'Tensordot_1/MatMul' lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] Original stack trace for 'Tensordot_1/MatMul': File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main true_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 312, in run _run_main(main, args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1496, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 5188, in tensordot ab_matmul = matmul(a_reshape, b_reshape) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 3794, in matmul return gen_math_ops.mat_mul( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/gen_math_ops.py", line 6035, in mat_mul _, _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 795, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 3383, in _create_op_internal ret = Operation.from_node_def( INFO:tensorflow:Searching within error strings: 'Matrix size-incompatible: In\[0\]: \[2,2\], In\[1\]: \[3,2\]' within 'Graph execution error: Detected at node 'Tensordot_1/MatMul' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) Node: 'Tensordot_1/MatMul' lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] Caused by: Tensordot_1/MatMul' I0421 04:00:49.519042 281473637577568 test_util.py:3533] Searching within error strings: 'Matrix size-incompatible: In\[0\]: \[2,2\], In\[1\]: \[3,2\]' within 'Graph execution error: Detected at node 'Tensordot_1/MatMul' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) Node: 'Tensordot_1/MatMul' lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] Caused by: Tensordot_1/MatMul' INFO:tensorflow:time(__main__.TensordotTest.test_invalid_shape): 0.63s I0421 04:00:49.521513 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_invalid_shape): 0.63s [ FAILED ] TensordotTest.test_invalid_shape [ RUN ] TensordotTest.test_test_tensordot_complex128_2_1_0_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:49.528336 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_2_1_0_True): 0.0s I0421 04:00:49.529656 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:49.530300 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_2_1_0_True): 0.0s I0421 04:00:49.530594 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex128_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_complex128_2_5_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:49.531211 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py:147: DeprecationWarning: This function is deprecated. Please call randint(1, 5 + 1) instead a_shape = np.random.random_integers(1, _MAXDIM, rank_a_) /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py:148: DeprecationWarning: This function is deprecated. Please call randint(1, 5 + 1) instead b_shape = np.random.random_integers(1, _MAXDIM, rank_b_) /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py:149: DeprecationWarning: This function is deprecated. Please call randint(1, 5 + 1) instead shared_shape = np.random.random_integers(1, _MAXDIM, num_dims_) INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_2_5_2_True): 0.47s I0421 04:00:50.001468 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_2_5_2_True): 0.47s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:50.004387 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_2_5_2_True): 0.0s I0421 04:00:50.004847 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex128_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_complex128_4_4_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:50.005537 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_4_4_4_True): 2.02s I0421 04:00:52.023304 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_4_4_4_True): 2.02s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:52.033186 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_4_4_4_True): 0.0s I0421 04:00:52.033723 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex128_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_complex128_5_2_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:52.034409 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_5_2_2_True): 0.39s I0421 04:00:52.427815 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_5_2_2_True): 0.39s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:52.430996 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_5_2_2_True): 0.0s I0421 04:00:52.431483 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex128_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_complex128_5_5_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:52.432180 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_5_5_4_True): 1.87s I0421 04:00:54.301138 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_5_5_4_True): 1.87s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:54.310513 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_5_5_4_True): 0.0s I0421 04:00:54.311034 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex128_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_complex64_2_1_0_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:54.311708 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_2_1_0_True): 0.0s I0421 04:00:54.312709 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:54.313262 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_2_1_0_True): 0.0s I0421 04:00:54.313534 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex64_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_complex64_2_5_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:54.314130 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_2_5_2_True): 0.42s I0421 04:00:54.731224 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_2_5_2_True): 0.42s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:54.734479 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_2_5_2_True): 0.0s I0421 04:00:54.734958 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex64_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_complex64_4_4_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:54.735659 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_4_4_4_True): 2.04s I0421 04:00:56.779764 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_4_4_4_True): 2.04s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:56.789925 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_4_4_4_True): 0.0s I0421 04:00:56.790504 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex64_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_complex64_5_2_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:56.791378 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_5_2_2_True): 0.43s I0421 04:00:57.218086 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_5_2_2_True): 0.43s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:57.221708 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_5_2_2_True): 0.0s I0421 04:00:57.222235 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex64_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_complex64_5_5_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:57.223001 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_5_5_4_True): 1.93s I0421 04:00:59.151590 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_5_5_4_True): 1.93s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:59.162033 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_5_5_4_True): 0.0s I0421 04:00:59.162673 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex64_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_float16_2_1_0_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:59.163482 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_2_1_0_True): 0.0s I0421 04:00:59.164661 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:59.165334 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_2_1_0_True): 0.0s I0421 04:00:59.165670 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float16_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_float16_2_5_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:59.166398 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_2_5_2_True): 0.44s I0421 04:00:59.604926 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_2_5_2_True): 0.44s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:00:59.608428 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_2_5_2_True): 0.0s I0421 04:00:59.608976 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float16_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_float16_4_4_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:00:59.609764 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_4_4_4_True): 1.9s I0421 04:01:01.512633 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_4_4_4_True): 1.9s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:01.523133 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_4_4_4_True): 0.0s I0421 04:01:01.523765 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float16_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_float16_5_2_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:01.524560 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_5_2_2_True): 0.43s I0421 04:01:01.957354 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_5_2_2_True): 0.43s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:01.960796 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_5_2_2_True): 0.0s I0421 04:01:01.961329 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float16_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_float16_5_5_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:01.962100 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_5_5_4_True): 1.88s I0421 04:01:03.844313 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_5_5_4_True): 1.88s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:03.853949 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_5_5_4_True): 0.0s I0421 04:01:03.854496 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float16_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_float32_2_1_0_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:03.855221 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_2_1_0_True): 0.0s I0421 04:01:03.856330 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:03.856928 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_2_1_0_True): 0.0s I0421 04:01:03.857218 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float32_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_float32_2_5_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:03.857846 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_2_5_2_True): 0.53s I0421 04:01:04.387913 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_2_5_2_True): 0.53s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:04.392089 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_2_5_2_True): 0.0s I0421 04:01:04.392504 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float32_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_float32_4_4_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:04.393117 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_4_4_4_True): 2.38s I0421 04:01:06.777401 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_4_4_4_True): 2.38s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:06.790698 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_4_4_4_True): 0.0s I0421 04:01:06.791179 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float32_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_float32_5_2_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:06.791815 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_5_2_2_True): 0.46s I0421 04:01:07.247739 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_5_2_2_True): 0.46s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:07.251870 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_5_2_2_True): 0.0s I0421 04:01:07.252285 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float32_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_float32_5_5_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:07.252897 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_5_5_4_True): 2.28s I0421 04:01:09.534465 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_5_5_4_True): 2.28s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:09.545730 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_5_5_4_True): 0.0s I0421 04:01:09.546214 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float32_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_float64_2_1_0_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:09.547010 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_2_1_0_True): 0.0s I0421 04:01:09.547972 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:09.548537 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_2_1_0_True): 0.0s I0421 04:01:09.548810 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float64_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_float64_2_5_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:09.549360 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_2_5_2_True): 0.4s I0421 04:01:09.949607 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_2_5_2_True): 0.4s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:09.953190 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_2_5_2_True): 0.0s I0421 04:01:09.953590 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float64_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_float64_4_4_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:09.954199 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_4_4_4_True): 1.77s I0421 04:01:11.725575 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_4_4_4_True): 1.77s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:11.735231 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_4_4_4_True): 0.0s I0421 04:01:11.735668 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float64_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_float64_5_2_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:11.736303 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_5_2_2_True): 0.39s I0421 04:01:12.123911 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_5_2_2_True): 0.39s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:12.127002 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_5_2_2_True): 0.0s I0421 04:01:12.127396 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float64_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_float64_5_5_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:12.127988 281473637577568 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_5_5_4_True): 1.87s I0421 04:01:13.997531 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_5_5_4_True): 1.87s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:14.007048 281473637577568 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_5_5_4_True): 0.0s I0421 04:01:14.007490 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float64_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex128_2_1_0_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:14.008090 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_1_0_True): 0.0s I0421 04:01:14.008910 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:14.009425 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_1_0_True): 0.0s I0421 04:01:14.009680 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex128_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex128_2_5_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:14.010209 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_5_2_True): 0.46s I0421 04:01:14.465239 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_5_2_True): 0.46s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:14.467162 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_5_2_True): 0.0s I0421 04:01:14.467510 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex128_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex128_4_4_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:14.468099 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_4_4_4_True): 1.54s I0421 04:01:16.011141 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_4_4_4_True): 1.54s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:16.013450 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_4_4_4_True): 0.0s I0421 04:01:16.013826 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex128_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex128_5_2_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:16.014417 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_2_2_True): 0.21s I0421 04:01:16.222389 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_2_2_True): 0.21s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:16.224180 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_2_2_True): 0.0s I0421 04:01:16.224565 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex128_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex128_5_5_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:16.225162 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_5_4_True): 1.88s I0421 04:01:18.108227 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_5_4_True): 1.88s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:18.110408 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_5_4_True): 0.0s I0421 04:01:18.110757 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex128_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex64_2_1_0_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:18.111377 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_1_0_True): 0.0s I0421 04:01:18.112185 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:18.112700 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_1_0_True): 0.0s I0421 04:01:18.112962 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex64_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex64_2_5_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:18.113502 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_5_2_True): 0.18s I0421 04:01:18.293207 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_5_2_True): 0.18s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:18.295145 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_5_2_True): 0.0s I0421 04:01:18.295498 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex64_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex64_4_4_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:18.296094 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_4_4_4_True): 1.63s I0421 04:01:19.923601 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_4_4_4_True): 1.63s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:19.926862 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_4_4_4_True): 0.0s I0421 04:01:19.927340 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex64_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex64_5_2_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:19.928011 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_2_2_True): 0.34s I0421 04:01:20.268040 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_2_2_True): 0.34s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:20.270255 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_2_2_True): 0.0s I0421 04:01:20.270664 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex64_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex64_5_5_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:20.271340 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_5_4_True): 1.61s I0421 04:01:21.877658 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_5_4_True): 1.61s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:21.880605 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_5_4_True): 0.0s I0421 04:01:21.881052 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex64_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float16_2_1_0_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:21.881731 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_1_0_True): 0.0s I0421 04:01:21.882791 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:21.883339 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_1_0_True): 0.0s I0421 04:01:21.883615 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float16_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float16_2_5_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:21.884204 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_5_2_True): 0.35s I0421 04:01:22.233429 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_5_2_True): 0.35s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:22.235733 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_5_2_True): 0.0s I0421 04:01:22.236239 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float16_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float16_4_4_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:22.236985 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_4_4_4_True): 1.05s I0421 04:01:23.287604 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_4_4_4_True): 1.05s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:23.290227 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_4_4_4_True): 0.0s I0421 04:01:23.290681 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float16_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float16_5_2_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:23.291378 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_2_2_True): 0.45s I0421 04:01:23.744708 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_2_2_True): 0.45s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:23.746788 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_2_2_True): 0.0s I0421 04:01:23.747217 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float16_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float16_5_5_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:23.748008 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_5_4_True): 0.43s I0421 04:01:24.182205 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_5_4_True): 0.43s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:24.185451 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_5_4_True): 0.0s I0421 04:01:24.185891 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float16_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float32_2_1_0_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:24.186641 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_1_0_True): 0.0s I0421 04:01:24.187695 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:24.692757 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_1_0_True): 0.0s I0421 04:01:24.693391 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float32_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float32_2_5_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:24.694431 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_5_2_True): 0.28s I0421 04:01:24.978733 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_5_2_True): 0.28s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:24.980892 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_5_2_True): 0.0s I0421 04:01:24.981357 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float32_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float32_4_4_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:24.982085 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_4_4_4_True): 1.33s I0421 04:01:26.307688 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_4_4_4_True): 1.33s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:26.310274 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_4_4_4_True): 0.0s I0421 04:01:26.310754 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float32_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float32_5_2_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:26.311470 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_2_2_True): 0.31s I0421 04:01:26.617171 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_2_2_True): 0.31s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:26.619155 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_2_2_True): 0.0s I0421 04:01:26.619530 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float32_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float32_5_5_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:26.620203 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_5_4_True): 1.39s I0421 04:01:28.012428 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_5_4_True): 1.39s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:28.015039 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_5_4_True): 0.0s I0421 04:01:28.015552 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float32_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float64_2_1_0_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:28.016360 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_1_0_True): 0.0s I0421 04:01:28.017524 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:28.018173 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_1_0_True): 0.0s I0421 04:01:28.018491 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float64_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float64_2_5_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:28.019169 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_5_2_True): 0.21s I0421 04:01:28.227752 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_5_2_True): 0.21s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:28.229886 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_5_2_True): 0.0s I0421 04:01:28.230404 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float64_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float64_4_4_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:28.231216 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_4_4_4_True): 1.0s I0421 04:01:29.231612 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_4_4_4_True): 1.0s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:29.234290 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_4_4_4_True): 0.0s I0421 04:01:29.234838 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float64_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float64_5_2_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:29.235651 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_2_2_True): 0.22s I0421 04:01:29.452601 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_2_2_True): 0.22s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:29.455091 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_2_2_True): 0.0s I0421 04:01:29.455637 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float64_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:01:29.456501 281473637577568 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_4_True): 1.51s I0421 04:01:30.963684 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_4_True): 1.51s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:01:30.966463 281473637577568 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_4_True): 0.0s I0421 04:01:30.966995 281473637577568 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_4_True ====================================================================== FAIL: test_invalid_shape (__main__.TensordotTest) TensordotTest.test_invalid_shape ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1379, in _do_call return fn(*args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1362, in _run_fn return self._call_tf_sessionrun(options, feed_dict, fetch_list, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1455, in _call_tf_sessionrun return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict, tensorflow.python.framework.errors_impl.InvalidArgumentError: lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 3538, in assertRaisesWithPredicateMatch yield File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 60, in test_invalid_shape _ = sess.run( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 2059, in run return super().run(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 969, in run result = self._run(None, fetches, feed_dict, options_ptr, tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error: Detected at node 'Tensordot_1/MatMul' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) Node: 'Tensordot_1/MatMul' lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] Original stack trace for 'Tensordot_1/MatMul': File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main true_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 312, in run _run_main(main, args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1496, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 5188, in tensordot ab_matmul = matmul(a_reshape, b_reshape) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 3794, in matmul return gen_math_ops.mat_mul( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/gen_math_ops.py", line 6035, in mat_mul _, _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 795, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 3383, in _create_op_internal ret = Operation.from_node_def( During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1496, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 60, in test_invalid_shape _ = sess.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 3542, in assertRaisesWithPredicateMatch raise AssertionError("Exception of type %s: %s" % AssertionError: Exception of type : Graph execution error: Detected at node 'Tensordot_1/MatMul' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) Node: 'Tensordot_1/MatMul' lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] Original stack trace for 'Tensordot_1/MatMul': File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main true_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 312, in run _run_main(main, args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1496, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 5188, in tensordot ab_matmul = matmul(a_reshape, b_reshape) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 3794, in matmul return gen_math_ops.mat_mul( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/gen_math_ops.py", line 6035, in mat_mul _, _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 795, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 3383, in _create_op_internal ret = Operation.from_node_def( ---------------------------------------------------------------------- Ran 51 tests in 42.081s FAILED (failures=1, skipped=50) ================================================================================ ==================== Test output for //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu (shard 2 of 20): 2023-04-21 04:01:33.070986: I tensorflow/core/util/port.cc:116] Experimental oneDNN custom operations are on. If you experience issues, please turn them off by setting the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. Running tests under Python 3.8.10: /usr/local/bin/python3 [ RUN ] TensordotTest.test_invalid_shape INFO:tensorflow:Running test_invalid_shape in GRAPH mode. I0421 04:01:35.048861 281473052930912 test_util.py:1492] Running test_invalid_shape in GRAPH mode. WARNING:tensorflow:From /usr/lib/python3.8/contextlib.py:83: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `self.session()` or `self.cached_session()` instead. W0421 04:01:35.049261 281473052930912 deprecation.py:364] From /usr/lib/python3.8/contextlib.py:83: TensorFlowTestCase.test_session (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version. Instructions for updating: Use `self.session()` or `self.cached_session()` instead. 2023-04-21 04:01:35.172200: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:375] MLIR V1 optimization pass is not enabled 2023-04-21 04:01:35.269084: I tensorflow/core/common_runtime/executor.cc:1210] [/job:localhost/replica:0/task:0/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] ERROR:tensorflow:Graph execution error: Detected at node 'Tensordot_1/MatMul' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) Node: 'Tensordot_1/MatMul' lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] Original stack trace for 'Tensordot_1/MatMul': File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main true_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 312, in run _run_main(main, args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1496, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 5188, in tensordot ab_matmul = matmul(a_reshape, b_reshape) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 3794, in matmul return gen_math_ops.mat_mul( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/gen_math_ops.py", line 6035, in mat_mul _, _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 795, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 3383, in _create_op_internal ret = Operation.from_node_def( E0421 04:01:35.874701 281473052930912 test_util.py:2065] Graph execution error: Detected at node 'Tensordot_1/MatMul' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) Node: 'Tensordot_1/MatMul' lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] Original stack trace for 'Tensordot_1/MatMul': File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main true_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 312, in run _run_main(main, args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1496, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 5188, in tensordot ab_matmul = matmul(a_reshape, b_reshape) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 3794, in matmul return gen_math_ops.mat_mul( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/gen_math_ops.py", line 6035, in mat_mul _, _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 795, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 3383, in _create_op_internal ret = Operation.from_node_def( INFO:tensorflow:Searching within error strings: 'Matrix size-incompatible: In\[0\]: \[2,2\], In\[1\]: \[3,2\]' within 'Graph execution error: Detected at node 'Tensordot_1/MatMul' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) Node: 'Tensordot_1/MatMul' lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] Caused by: Tensordot_1/MatMul' I0421 04:01:35.875084 281473052930912 test_util.py:3533] Searching within error strings: 'Matrix size-incompatible: In\[0\]: \[2,2\], In\[1\]: \[3,2\]' within 'Graph execution error: Detected at node 'Tensordot_1/MatMul' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) Node: 'Tensordot_1/MatMul' lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] Caused by: Tensordot_1/MatMul' INFO:tensorflow:time(__main__.TensordotTest.test_invalid_shape): 0.83s I0421 04:01:35.877974 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_invalid_shape): 0.83s [ FAILED ] TensordotTest.test_invalid_shape [ RUN ] TensordotTest.test_test_tensordot_complex128_2_1_0_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:35.886100 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_2_1_0_True): 0.0s I0421 04:01:35.887405 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:35.888070 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_2_1_0_True): 0.0s I0421 04:01:35.888381 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex128_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_complex128_2_5_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:35.888967 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py:147: DeprecationWarning: This function is deprecated. Please call randint(1, 5 + 1) instead a_shape = np.random.random_integers(1, _MAXDIM, rank_a_) /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py:148: DeprecationWarning: This function is deprecated. Please call randint(1, 5 + 1) instead b_shape = np.random.random_integers(1, _MAXDIM, rank_b_) /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py:149: DeprecationWarning: This function is deprecated. Please call randint(1, 5 + 1) instead shared_shape = np.random.random_integers(1, _MAXDIM, num_dims_) INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_2_5_2_True): 0.43s I0421 04:01:36.321790 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_2_5_2_True): 0.43s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:36.324467 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_2_5_2_True): 0.0s I0421 04:01:36.324854 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex128_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_complex128_4_4_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:36.325452 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_4_4_4_True): 1.78s I0421 04:01:38.109381 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_4_4_4_True): 1.78s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:38.118782 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_4_4_4_True): 0.0s I0421 04:01:38.119283 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex128_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_complex128_5_2_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:38.119886 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_5_2_2_True): 0.39s I0421 04:01:38.513543 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_5_2_2_True): 0.39s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:38.517858 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_5_2_2_True): 0.0s I0421 04:01:38.518416 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex128_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_complex128_5_5_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:38.519127 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_5_5_4_True): 1.89s I0421 04:01:40.410948 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_5_5_4_True): 1.89s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:40.423219 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex128_5_5_4_True): 0.0s I0421 04:01:40.423766 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex128_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex128_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_complex64_2_1_0_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:40.424476 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_2_1_0_True): 0.0s I0421 04:01:40.425565 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:40.426149 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_2_1_0_True): 0.0s I0421 04:01:40.426434 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex64_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_complex64_2_5_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:40.427036 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_2_5_2_True): 0.45s I0421 04:01:40.880564 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_2_5_2_True): 0.45s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:40.884763 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_2_5_2_True): 0.0s I0421 04:01:40.885323 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex64_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_complex64_4_4_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:40.886059 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_4_4_4_True): 2.36s I0421 04:01:43.245181 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_4_4_4_True): 2.36s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:43.260630 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_4_4_4_True): 0.0s I0421 04:01:43.261198 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex64_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_complex64_5_2_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:43.261926 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_5_2_2_True): 0.42s I0421 04:01:43.678287 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_5_2_2_True): 0.42s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:43.682881 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_5_2_2_True): 0.0s I0421 04:01:43.683421 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex64_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_complex64_5_5_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:43.684167 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_5_5_4_True): 1.85s I0421 04:01:45.531106 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_5_5_4_True): 1.85s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:45.542727 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_complex64_5_5_4_True): 0.0s I0421 04:01:45.543228 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_complex64_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_complex64_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_float16_2_1_0_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:45.544029 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_2_1_0_True): 0.0s I0421 04:01:45.545031 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:45.545598 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_2_1_0_True): 0.0s I0421 04:01:45.545872 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float16_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_float16_2_5_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:45.546457 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_2_5_2_True): 0.48s I0421 04:01:46.027612 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_2_5_2_True): 0.48s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:46.030841 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_2_5_2_True): 0.0s I0421 04:01:46.031217 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float16_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_float16_4_4_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:46.031807 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_4_4_4_True): 1.78s I0421 04:01:47.809774 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_4_4_4_True): 1.78s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:47.820842 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_4_4_4_True): 0.0s I0421 04:01:47.821341 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float16_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_float16_5_2_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:47.822219 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_5_2_2_True): 0.4s I0421 04:01:48.219328 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_5_2_2_True): 0.4s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:48.222829 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_5_2_2_True): 0.0s I0421 04:01:48.223251 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float16_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_float16_5_5_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:48.223865 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_5_5_4_True): 1.89s I0421 04:01:50.112700 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_5_5_4_True): 1.89s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:50.123590 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float16_5_5_4_True): 0.0s I0421 04:01:50.124080 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float16_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float16_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_float32_2_1_0_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:50.124687 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_2_1_0_True): 0.0s I0421 04:01:50.125668 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:50.126222 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_2_1_0_True): 0.0s I0421 04:01:50.126496 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float32_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_float32_2_5_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:50.127030 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_2_5_2_True): 0.68s I0421 04:01:50.805169 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_2_5_2_True): 0.68s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:50.808593 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_2_5_2_True): 0.0s I0421 04:01:50.808980 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float32_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_float32_4_4_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:50.809559 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_4_4_4_True): 2.69s I0421 04:01:53.499597 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_4_4_4_True): 2.69s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:53.510766 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_4_4_4_True): 0.0s I0421 04:01:53.511232 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float32_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_float32_5_2_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:53.511941 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_5_2_2_True): 0.48s I0421 04:01:53.995157 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_5_2_2_True): 0.48s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:53.998848 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_5_2_2_True): 0.0s I0421 04:01:53.999284 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float32_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_float32_5_5_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:53.999908 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_5_5_4_True): 2.16s I0421 04:01:56.162781 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_5_5_4_True): 2.16s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:56.172878 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float32_5_5_4_True): 0.0s I0421 04:01:56.173364 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float32_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float32_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_float64_2_1_0_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:56.173981 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_2_1_0_True): 0.0s I0421 04:01:56.174950 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:56.175489 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_2_1_0_True): 0.0s I0421 04:01:56.175759 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float64_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_float64_2_5_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:56.176356 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_2_5_2_True): 0.41s I0421 04:01:56.586854 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_2_5_2_True): 0.41s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:56.591381 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_2_5_2_True): 0.0s I0421 04:01:56.591894 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float64_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_float64_4_4_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:56.592608 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_4_4_4_True): 1.99s I0421 04:01:58.585849 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_4_4_4_True): 1.99s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:58.594935 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_4_4_4_True): 0.0s I0421 04:01:58.595418 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float64_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_float64_5_2_2_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:58.596014 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_5_2_2_True): 0.4s I0421 04:01:58.998498 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_5_2_2_True): 0.4s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:01:59.001715 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_5_2_2_True): 0.0s I0421 04:01:59.002139 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float64_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_float64_5_5_4_True INFO:tensorflow:Running test_tensordot in GRAPH mode. I0421 04:01:59.002820 281473052930912 test_util.py:1492] Running test_tensordot in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_5_5_4_True): 2.02s I0421 04:02:01.018155 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_5_5_4_True): 2.02s INFO:tensorflow:Running test_tensordot in EAGER mode. I0421 04:02:01.032473 281473052930912 test_util.py:1501] Running test_tensordot in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_float64_5_5_4_True): 0.0s I0421 04:02:01.033028 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_float64_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_float64_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex128_2_1_0_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:01.033740 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_1_0_True): 0.0s I0421 04:02:01.034817 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:01.035400 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_1_0_True): 0.0s I0421 04:02:01.035670 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex128_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex128_2_5_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:01.036345 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_5_2_True): 0.19s I0421 04:02:01.221677 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_5_2_True): 0.19s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:01.224251 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_5_2_True): 0.0s I0421 04:02:01.224687 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex128_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex128_4_4_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:01.225364 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_4_4_4_True): 0.94s I0421 04:02:02.170329 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_4_4_4_True): 0.94s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:02.173655 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_4_4_4_True): 0.0s I0421 04:02:02.174140 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex128_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex128_5_2_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:02.174834 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_2_2_True): 0.18s I0421 04:02:02.354596 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_2_2_True): 0.18s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:02.356866 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_2_2_True): 0.0s I0421 04:02:02.357328 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex128_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex128_5_5_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:02.358023 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_5_4_True): 1.35s I0421 04:02:03.710177 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_5_4_True): 1.35s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:03.712207 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_5_4_True): 0.0s I0421 04:02:03.712544 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex128_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex128_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex64_2_1_0_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:03.713152 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_1_0_True): 0.0s I0421 04:02:03.714001 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:03.714512 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_1_0_True): 0.0s I0421 04:02:03.714770 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex64_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex64_2_5_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:03.715304 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_5_2_True): 0.27s I0421 04:02:03.988133 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_5_2_True): 0.27s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:03.990275 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_5_2_True): 0.0s I0421 04:02:03.990636 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex64_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex64_4_4_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:03.991214 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_4_4_4_True): 1.22s I0421 04:02:05.213711 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_4_4_4_True): 1.22s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:05.215941 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_4_4_4_True): 0.0s I0421 04:02:05.216337 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex64_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex64_5_2_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:05.216920 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_2_2_True): 0.17s I0421 04:02:05.387743 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_2_2_True): 0.17s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:05.389517 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_2_2_True): 0.0s I0421 04:02:05.389881 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex64_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_complex64_5_5_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:05.390482 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_5_4_True): 0.88s I0421 04:02:06.267693 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_5_4_True): 0.88s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:06.269902 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_5_4_True): 0.0s I0421 04:02:06.270266 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_complex64_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_complex64_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float16_2_1_0_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:06.270841 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_1_0_True): 0.0s I0421 04:02:06.271766 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:06.272287 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_1_0_True): 0.0s I0421 04:02:06.272547 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float16_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float16_2_5_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:06.273086 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_5_2_True): 0.17s I0421 04:02:06.442909 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_5_2_True): 0.17s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:06.444517 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_5_2_True): 0.0s I0421 04:02:06.444849 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float16_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float16_4_4_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:06.445424 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_4_4_4_True): 0.33s I0421 04:02:06.777113 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_4_4_4_True): 0.33s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:06.779042 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_4_4_4_True): 0.0s I0421 04:02:06.779378 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float16_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float16_5_2_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:06.779954 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_2_2_True): 0.29s I0421 04:02:07.070694 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_2_2_True): 0.29s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:07.072123 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_2_2_True): 0.0s I0421 04:02:07.072404 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float16_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float16_5_5_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:07.072966 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_5_4_True): 0.34s I0421 04:02:07.417800 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_5_4_True): 0.34s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:07.420320 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_5_4_True): 0.0s I0421 04:02:07.420690 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float16_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float16_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float32_2_1_0_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:07.421276 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_1_0_True): 0.0s I0421 04:02:07.422220 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:07.716954 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_1_0_True): 0.0s I0421 04:02:07.717387 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float32_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float32_2_5_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:07.718130 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_5_2_True): 0.7s I0421 04:02:08.415624 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_5_2_True): 0.7s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:08.417146 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_5_2_True): 0.0s I0421 04:02:08.417441 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float32_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float32_4_4_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:08.417989 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_4_4_4_True): 0.69s I0421 04:02:09.104067 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_4_4_4_True): 0.69s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:09.106900 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_4_4_4_True): 0.0s I0421 04:02:09.107340 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float32_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float32_5_2_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:09.108005 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_2_2_True): 0.18s I0421 04:02:09.291249 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_2_2_True): 0.18s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:09.293427 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_2_2_True): 0.0s I0421 04:02:09.293835 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float32_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float32_5_5_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:09.294492 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_5_4_True): 0.81s I0421 04:02:10.105438 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_5_4_True): 0.81s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:10.108356 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_5_4_True): 0.0s I0421 04:02:10.108827 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float32_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float32_5_5_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float64_2_1_0_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:10.109516 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_1_0_True): 0.0s I0421 04:02:10.110650 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_1_0_True): 0.0s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:10.111215 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_1_0_True): 0.0s I0421 04:02:10.111481 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_1_0_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float64_2_1_0_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float64_2_5_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:10.112079 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_5_2_True): 0.19s I0421 04:02:10.298122 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_5_2_True): 0.19s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:10.300297 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_5_2_True): 0.0s I0421 04:02:10.300676 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_2_5_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float64_2_5_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float64_4_4_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:10.301338 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_4_4_4_True): 0.73s I0421 04:02:11.034271 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_4_4_4_True): 0.73s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:11.036389 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_4_4_4_True): 0.0s I0421 04:02:11.036760 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_4_4_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float64_4_4_4_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float64_5_2_2_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:11.037346 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_2_2_True): 0.18s I0421 04:02:11.213445 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_2_2_True): 0.18s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:11.215213 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_2_2_True): 0.0s I0421 04:02:11.215559 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_2_2_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float64_5_2_2_True [ RUN ] TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_4_True INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode. I0421 04:02:11.216160 281473052930912 test_util.py:1492] Running test_tensordot_scalar_axes in GRAPH mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_4_True): 0.59s I0421 04:02:11.803732 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_4_True): 0.59s INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode. I0421 04:02:11.805991 281473052930912 test_util.py:1501] Running test_tensordot_scalar_axes in EAGER mode. INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_4_True): 0.0s I0421 04:02:11.806393 281473052930912 test_util.py:2462] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_4_True): 0.0s [ SKIPPED ] TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_4_True ====================================================================== FAIL: test_invalid_shape (__main__.TensordotTest) TensordotTest.test_invalid_shape ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1379, in _do_call return fn(*args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1362, in _run_fn return self._call_tf_sessionrun(options, feed_dict, fetch_list, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1455, in _call_tf_sessionrun return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict, tensorflow.python.framework.errors_impl.InvalidArgumentError: lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 3538, in assertRaisesWithPredicateMatch yield File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 60, in test_invalid_shape _ = sess.run( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 2059, in run return super().run(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 969, in run result = self._run(None, fetches, feed_dict, options_ptr, tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error: Detected at node 'Tensordot_1/MatMul' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) Node: 'Tensordot_1/MatMul' lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] Original stack trace for 'Tensordot_1/MatMul': File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main true_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 312, in run _run_main(main, args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1496, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 5188, in tensordot ab_matmul = matmul(a_reshape, b_reshape) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 3794, in matmul return gen_math_ops.mat_mul( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/gen_math_ops.py", line 6035, in mat_mul _, _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 795, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 3383, in _create_op_internal ret = Operation.from_node_def( During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1496, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 60, in test_invalid_shape _ = sess.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 3542, in assertRaisesWithPredicateMatch raise AssertionError("Exception of type %s: %s" % AssertionError: Exception of type : Graph execution error: Detected at node 'Tensordot_1/MatMul' defined at (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) Node: 'Tensordot_1/MatMul' lhs mismatch rhs shape: 2 vs. 3: [2,2] [3,2] 0 0 [[{{node Tensordot_1/MatMul}}]] Original stack trace for 'Tensordot_1/MatMul': File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 248, in test_lib.main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/test.py", line 56, in main return _googletest.main(argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 62, in main benchmark.benchmarks_main(true_main=main_wrapper, argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/benchmark.py", line 489, in benchmarks_main true_main() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 60, in main_wrapper return app.run(main=g_main, argv=args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 312, in run _run_main(main, args) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/platform/googletest.py", line 51, in g_main absltest_main(argv=argv) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2049, in main _run_in_app(run_tests, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2156, in _run_in_app function(argv, args, kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2568, in run_tests result = _run_and_get_tests_result( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/absl_py/absl/testing/absltest.py", line 2537, in _run_and_get_tests_result test_program = unittest.TestProgram(*args, **kwargs) File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__ self.runTests() File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/usr/lib/python3.8/unittest/runner.py", line 176, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/suite.py", line 122, in run test(result) File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__ return self.run(*args, **kwds) File "/usr/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1496, in decorated f(self, *args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 59, in test_invalid_shape output = math_ops.tensordot(a_ph, b_ph, axes_ph) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 5188, in tensordot ab_matmul = matmul(a_reshape, b_reshape) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/traceback_utils.py", line 141, in error_handler return fn(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler return dispatch_target(*args, **kwargs) File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/math_ops.py", line 3794, in matmul return gen_math_ops.mat_mul( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/ops/gen_math_ops.py", line 6035, in mat_mul _, _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 795, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 3383, in _create_op_internal ret = Operation.from_node_def( ---------------------------------------------------------------------- Ran 51 tests in 36.759s FAILED (failures=1, skipped=50) ================================================================================ //tensorflow/c:c_api_experimental_test PASSED in 32.2s //tensorflow/c:c_api_function_test PASSED in 27.6s //tensorflow/c:c_api_test_cpu PASSED in 31.7s //tensorflow/c:c_test PASSED in 30.2s //tensorflow/c:env_test_cpu PASSED in 25.0s //tensorflow/c:kernels_test_cpu PASSED in 42.3s //tensorflow/c:ops_test PASSED in 29.4s //tensorflow/c:while_loop_test PASSED in 26.8s //tensorflow/c/eager:c_api_cluster_test_cpu PASSED in 31.9s //tensorflow/c/eager:c_api_remote_function_test_cpu PASSED in 29.8s //tensorflow/c/eager:c_api_remote_test_cpu PASSED in 29.0s //tensorflow/c/eager:custom_device_test PASSED in 27.7s //tensorflow/c/eager/parallel_device:parallel_device_lib_test PASSED in 30.2s //tensorflow/c/eager/parallel_device:parallel_device_remote_test PASSED in 38.2s //tensorflow/c/eager/parallel_device:parallel_device_test PASSED in 26.8s //tensorflow/c/experimental/filesystem/plugins/gcs:expiring_lru_cache_test PASSED in 0.2s //tensorflow/c/experimental/filesystem/plugins/gcs:ram_file_block_cache_test PASSED in 2.2s //tensorflow/c/experimental/grappler:grappler_test PASSED in 24.4s //tensorflow/c/experimental/ops/gen/common:case_format_test PASSED in 0.5s //tensorflow/c/experimental/ops/gen/cpp:cpp_generator_test PASSED in 0.7s //tensorflow/c/experimental/ops/gen/cpp/renderers:renderer_test PASSED in 0.8s //tensorflow/c/experimental/saved_model/core:constant_loading_test PASSED in 18.6s //tensorflow/c/experimental/saved_model/core:object_graph_traversal_test PASSED in 15.6s //tensorflow/c/experimental/saved_model/core:saved_variable_loading_test PASSED in 20.1s //tensorflow/c/experimental/saved_model/core:signature_flattening_test PASSED in 13.6s //tensorflow/c/experimental/saved_model/core:tf_concrete_function_loading_test PASSED in 16.2s //tensorflow/c/experimental/saved_model/core/ops:restore_ops_test PASSED in 12.7s //tensorflow/c/experimental/saved_model/core/ops:variable_ops_test PASSED in 15.6s //tensorflow/c/experimental/saved_model/internal:saved_model_api_test PASSED in 28.1s //tensorflow/c/experimental/stream_executor:stream_executor_test PASSED in 0.4s //tensorflow/c/kernels:bitcast_op_test PASSED in 0.6s //tensorflow/c/kernels:summary_op_benchmark_test PASSED in 0.6s //tensorflow/c/kernels:summary_op_test PASSED in 0.7s //tensorflow/c/kernels:tensor_shape_utils_test PASSED in 0.2s //tensorflow/cc:cc_op_gen_test PASSED in 0.2s //tensorflow/cc:client_client_session_test PASSED in 2.3s //tensorflow/cc:coordinator_test PASSED in 3.9s //tensorflow/cc:framework_cc_ops_test PASSED in 3.4s //tensorflow/cc:framework_gradient_checker_test PASSED in 2.7s //tensorflow/cc:framework_gradients_test PASSED in 5.3s //tensorflow/cc:framework_scope_test PASSED in 0.7s //tensorflow/cc:framework_while_gradients_test PASSED in 3.3s //tensorflow/cc:gradients_array_grad_test PASSED in 14.0s //tensorflow/cc:gradients_data_flow_grad_test PASSED in 2.3s //tensorflow/cc:gradients_functional_grad_test PASSED in 2.2s //tensorflow/cc:gradients_image_grad_test PASSED in 5.9s //tensorflow/cc:gradients_linalg_grad_test PASSED in 3.5s //tensorflow/cc:gradients_manip_grad_test PASSED in 2.0s //tensorflow/cc:gradients_math_grad_test PASSED in 11.8s //tensorflow/cc:gradients_nn_grad_test PASSED in 15.0s //tensorflow/cc:gradients_resource_variable_grad_test PASSED in 2.3s //tensorflow/cc:ops_const_op_test PASSED in 0.6s //tensorflow/cc:ops_while_loop_test PASSED in 2.1s //tensorflow/cc:queue_runner_test PASSED in 14.5s //tensorflow/cc/experimental/base/tests:tensor_test PASSED in 0.3s //tensorflow/cc/experimental/base/tests:tensorhandle_test PASSED in 31.0s //tensorflow/cc/experimental/libexport:load_test PASSED in 0.2s //tensorflow/cc/experimental/libexport:save_test PASSED in 0.1s //tensorflow/cc/experimental/libtf:libtf_module_test PASSED in 28.1s //tensorflow/cc/experimental/libtf:libtf_object_test PASSED in 0.1s //tensorflow/cc/experimental/libtf:libtf_perf_test PASSED in 0.1s //tensorflow/cc/experimental/libtf:libtf_runtime_test PASSED in 29.7s //tensorflow/cc/experimental/libtf:libtf_transform_test PASSED in 33.0s //tensorflow/cc/experimental/libtf:libtf_value_test PASSED in 0.1s //tensorflow/cc/experimental/libtf:libtf_visit_test PASSED in 0.2s //tensorflow/cc/experimental/libtf/impl:iostream_test PASSED in 0.1s //tensorflow/cc/experimental/libtf/impl:none_test PASSED in 0.1s //tensorflow/cc/experimental/libtf/impl:scalars_test PASSED in 0.3s //tensorflow/cc/experimental/libtf/impl:string_test PASSED in 0.3s //tensorflow/cc/experimental/libtf/impl:tensor_spec_test PASSED in 0.2s //tensorflow/cc/saved_model:bundle_v2_test PASSED in 0.1s //tensorflow/cc/saved_model:fingerprinting_test PASSED in 1.1s //tensorflow/cc/saved_model:metrics_test PASSED in 0.1s //tensorflow/cc/saved_model:reader_test PASSED in 0.1s //tensorflow/cc/saved_model:saved_model_bundle_lite_test PASSED in 7.4s //tensorflow/cc/saved_model:saved_model_bundle_test PASSED in 6.4s //tensorflow/cc/saved_model:util_test PASSED in 0.4s //tensorflow/cc/saved_model/experimental/tests:saved_model_api_test PASSED in 26.9s //tensorflow/cc/tools:freeze_saved_model_test PASSED in 2.1s //tensorflow/compiler/aot:codegen_test PASSED in 27.1s //tensorflow/compiler/jit:compilability_check_util_test PASSED in 17.3s //tensorflow/compiler/jit:deadness_analysis_test PASSED in 9.2s //tensorflow/compiler/jit:device_compilation_cache_test PASSED in 5.6s //tensorflow/compiler/jit:device_compilation_cluster_signature_test PASSED in 5.8s //tensorflow/compiler/jit:device_compilation_profiler_test PASSED in 29.9s //tensorflow/compiler/jit:device_compiler_client_test PASSED in 5.1s //tensorflow/compiler/jit:device_compiler_disable_test PASSED in 19.2s //tensorflow/compiler/jit:device_executable_persistor_test PASSED in 19.8s //tensorflow/compiler/jit:device_util_test PASSED in 4.8s //tensorflow/compiler/jit:encapsulate_util_test PASSED in 0.7s //tensorflow/compiler/jit:node_matchers_test PASSED in 0.7s //tensorflow/compiler/jit:resource_operation_safety_analysis_test PASSED in 9.2s //tensorflow/compiler/jit:shape_inference_test PASSED in 0.7s //tensorflow/compiler/jit:xla_activity_listener_test PASSED in 24.7s //tensorflow/compiler/jit:xla_cluster_util_test PASSED in 8.6s //tensorflow/compiler/jit:xla_compile_util_test PASSED in 5.4s //tensorflow/compiler/jit:xla_kernel_creator_test PASSED in 7.7s //tensorflow/compiler/jit:xla_launch_util_test PASSED in 21.8s //tensorflow/compiler/jit/tests:auto_clustering_test PASSED in 33.7s //tensorflow/compiler/mlir:mlir_graph_optimization_pass_test PASSED in 12.6s //tensorflow/compiler/mlir:register_common_dialects_test PASSED in 16.6s //tensorflow/compiler/mlir/lite:lstm_utils_test PASSED in 0.6s //tensorflow/compiler/mlir/lite:perception_ops_utils_test PASSED in 0.5s //tensorflow/compiler/mlir/lite:size_utils_test PASSED in 0.2s //tensorflow/compiler/mlir/lite:tftext_utils_test PASSED in 0.6s //tensorflow/compiler/mlir/lite/experimental/remat:rematerializer_test PASSED in 1.0s //tensorflow/compiler/mlir/lite/experimental/tac:execution_metadata_exporter_test PASSED in 3.2s //tensorflow/compiler/mlir/lite/experimental/tac/tests:compute-cost.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-gpu.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:device-transform-nnapi.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:fold-constants-to-subgraph.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:get-alternative-subgraph.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/experimental/tac/tests:get-op-cost.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/experimental/tac/tests:pick-subgraphs.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/experimental/tac/tests:raise-target-subgraphs.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/lite/experimental/tac/tests:target-annotation.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:device-transform-nnapi.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/experimental/tac/tests/e2e:simple-graph.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/metrics:error_collector_inst_test PASSED in 0.3s //tensorflow/compiler/mlir/lite/quantization:numerical_utils_test PASSED in 0.3s //tensorflow/compiler/mlir/lite/quantization/lite:quantize_model_test PASSED in 10.4s //tensorflow/compiler/mlir/lite/quantization/lite:quantize_weights_test PASSED in 10.4s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_default.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:fallback_to_flex_ops_legacy.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/quantization/tensorflow/tests:tf_to_quant_4bit.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/quantization/tests:import_quant_stats.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/sparsity:sparsify_model_test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:fold_broadcast.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:fuse_mhlo_convolution.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-inplaceupdate.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-skip-quantization-ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tf-fb-tf.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-add.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-broadcast_in_dim.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-clamp.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-compare.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-concat.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-constant.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-conv.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-dot.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-gather.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-max.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-mul.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-pad.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-reshape.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-rsqrt.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-scatter.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl-sub.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-stablehlo-tfl.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-add.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-broadcast.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-clamp.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-concat.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-constant.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-conv.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-max.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-mul.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-pad.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-reshape.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-rsqrt.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo-sub.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:legalize-tfl-stablehlo.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-allow-tf.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:odml-to-stablehlo-smuggle-resize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:optimize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-clamp.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-concat.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-conv.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-division.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-logistic.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-multiply.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-reduce-window.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-resize-bilinear.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo-subtract.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-serialize-stablehlo.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/stablehlo/tests:tf-tfl-translate-tf-quantize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/stablehlo/tests:unfuse_mhlo_batch_norm.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:analyze-variables.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:canonicalize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:const-fold.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:decompose-hybrid-quantization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:default_quant_params.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:dilated-conv.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:fuse-tftext.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests:get-arithmetic-count.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:guarantee_func_has_one_use.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:inlining.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:insert_call_once_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:legalize-tf-assert.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:legalize-tf-hashtables.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:legalize-tf-no-runtime-verification.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:legalize-tf-variables.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests:legalize-tf-while.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:legalize-tf.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/lite/tests:legalize_jax_random.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:lift_tflite_flex_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-default-to-single-batch.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list-enable-dynamic-update-slice.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:lower-static-tensor-list.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:modify_io_nodes.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:ops.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:optimize-after-quantization.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:optimize.mlir.test PASSED in 3.0s //tensorflow/compiler/mlir/lite/tests:optimize_functional_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:optimize_no_verify.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:optimize_op_order.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:partitioned-topological-sort.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:pin-ops-with-side-effects.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:post-quantize-dynamic-range.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:post-quantize.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:prepare-composite-functions-tf.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-dynamic-range.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training-16bits.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-post-training.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:prepare-quantize-signed.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests:prepare-quantize.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant-4bit.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:prepare-tf-fake-quant.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests:prepare-tf-with-allowing-bf16-and-f16-type-legalization.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests:prepare-tf.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:quantize-dynamic-range.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests:quantize-numeric-verify.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:quantize-variables.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests:quantize.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:raise-custom-ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:reduce_while_operands.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests:shape-inference.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:split-merged-operands.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:tfl_while_op_licm.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:tfl_while_outline.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests:trim-functions-tf.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests:unfold-large-splat-constant.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.line.part.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/debuginfo:v1_1.0_224_frozen.wrong_attr.stack.part.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/end2end:add.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:back2back_fake_quant.pbtxt.test PASSED in 1.7s //tensorflow/compiler/mlir/lite/tests/end2end:control_flow_v1.pbtxt.test PASSED in 1.2s //tensorflow/compiler/mlir/lite/tests/end2end:conv_2d.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:conv_2d_nchw.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/end2end:custom_opdef.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/end2end:disallow_stateful_partitioned_call.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_per_channel_4bit.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/end2end:fake_quant_without_identity_4bit.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/end2end:graph-input-node.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:graph_with_placeholder_with_default.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/end2end:if_op.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/end2end:quant_stats.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/end2end:unroll_batch_matmul_disabled.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:basic_lstm.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:bucketize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:constants.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:control_edges.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:custom_op.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:dynamic_shape.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:external_constant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:if_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:import_json.json.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:importer_test_min_max.cc.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_arrays.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:input_output_names_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:legacy_reshape.json.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.json.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:lstm.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:many_attribute_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:math.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:matmul.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:multi_output_op.json.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:optional_input.json.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:output_arrays.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:pruning_function_input_as_output.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quant_stats.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:quantization.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:reshape.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:signature_with_multiple_entry_points.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:simple.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:tf_variant_type.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_function_output.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:unranked_tensor.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir:while_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2exec:tfl_while_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:basic_lstm.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:bucketize.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:custom_op_with_tflite_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:depthwise_conv2d_v2.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_builtin.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_custom.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:disable_flex_enable_builtin.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:dynamic_shape_constant.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fake_quant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_exclusively.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_complex128.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_f64.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:flex_op_with_tflite_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:fully_connected_v2.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:hashtable_resource.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:if_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:logical.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:low_bit_packing.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_asym_attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:lstm_quantized.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:math.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:metadata.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v2.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:mul_v3.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:nn.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:numeric_verify.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:optional.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:quantization.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:reshape.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_output_override.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_multiple_entry_points.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:signature_def_with_no_inputs.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_connected_control_nodes.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:simple_with_unconnected_control_nodes.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:svdf_v2.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tf_entry_function.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:tfl_while_op.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:transpose_conv_optional.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:type_attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_lstm.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unidirectional_sequence_rnn.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unranked_tensor.mlir.test PASSED in 3.1s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:unsorted_segment_prod.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_func.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:variant_type_on_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/lite/tests/mlir2flatbuffer:while_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:calibrator_singleton_test PASSED in 0.2s //tensorflow/compiler/mlir/quantization/tensorflow/calibrator:custom_aggregator_op_test PASSED in 15.8s //tensorflow/compiler/mlir/quantization/tensorflow/cc:const_op_size_test PASSED in 0.5s //tensorflow/compiler/mlir/quantization/tensorflow/cc:convert_asset_args_test PASSED in 8.6s //tensorflow/compiler/mlir/quantization/tensorflow/cc:save_variables_test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/cc:status_macro_test PASSED in 0.3s //tensorflow/compiler/mlir/quantization/tensorflow/debugging:mlir_dump_test PASSED in 0.2s //tensorflow/compiler/mlir/quantization/tensorflow/python:concurrency_test PASSED in 32.9s //tensorflow/compiler/mlir/quantization/tensorflow/python:pywrap_quantize_model_test PASSED in 17.0s //tensorflow/compiler/mlir/quantization/tensorflow/python:representative_dataset_test PASSED in 8.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:cast_bf16_ops_to_f32.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_custom_aggregation_op_to_quant_stats.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_fake_quant_to_qdq.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_tf_quant_ops_to_mhlo.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:convert_tpu_model_to_cpu.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:duplicate_shape_determining_constants.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_flow.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:fake_quant_e2e_xla.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_custom_aggregation_ops.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_main_function.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_drq.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_quantized_functions_weight_only.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_restore_op.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/quantization/tensorflow/tests:insert_save_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:issue_ids_of_custom_aggregation_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_drq_min_elements.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:lift_quantizable_spots_as_functions_xla.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:mark_functions_noinline.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_initializer_function_ops_to_main.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:merge_save_function_ops_to_main.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:optimize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_lifting.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_drq_per_channel.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:prepare_quantize_ptq_per_channel.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:preprocess_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_drq.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_weight_only.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_composite_functions_xla.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_drq.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:quantize_xla.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:remove_var_init_by_const.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/quantization/tensorflow/tests:replace_cast_hacks_with_tf_xla_ops_large_constants.mlir.test PASSED in 13.4s //tensorflow/compiler/mlir/quantization/tensorflow/tests:unfreeze_constants.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/quantization/tensorflow/utils:tf_to_xla_attribute_utils_test PASSED in 31.9s //tensorflow/compiler/mlir/tensorflow:bridge_logger_test PASSED in 5.8s //tensorflow/compiler/mlir/tensorflow:cluster_util_test PASSED in 0.3s //tensorflow/compiler/mlir/tensorflow:convert_tensor_test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow:convert_type_test PASSED in 0.3s //tensorflow/compiler/mlir/tensorflow:device_util_test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow:dump_graph_test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow:dump_mlir_util_test PASSED in 9.2s //tensorflow/compiler/mlir/tensorflow:error_util_test PASSED in 0.2s //tensorflow/compiler/mlir/tensorflow:tf_saved_model_test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow:tpu_rewrite_device_util_test PASSED in 0.4s //tensorflow/compiler/mlir/tensorflow/tests:add_functions_for_exported_names.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:annotate-parameter-replication.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:batchmatmul_to_einsum.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:breakup-islands.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:cannonicalize_ops_outside_compilation.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:canonicalize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:canonicalize_compile_and_replicate_attributes.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:check_control_dependencies.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:cluster_formation.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:cluster_ops_by_policy.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:cluster_outlining.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:cluster_tf_ops_pass.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:constant-fold.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:constant_op_device_assignment.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:convert-tf-control-flow-to-scf.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:convert_control_to_data_outputs.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:convert_launch_func_to_tf_call.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:convert_session_initializer_to_function.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:convert_to_legacy_compile_and_replicate_attributes.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:decompose_reduce_dataset.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:decompose_resource_ops.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:device_assignment.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:device_assignment_by_func_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:device_attribute_to_launch.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:device_canonicalize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:device_copy.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:drop_while_shape_invariant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:einsum.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:embedding_pipelining.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:empty-main.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:end-to-end-tpu-reshard-variables.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests:executor_canonicalize.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:executor_island_coarsening.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:executor_island_materialize_const.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:extract_head_tail_outside_compilation.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:extract_outside_compilation.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:extract_tpu_copy_with_dynamic_shape_op.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:fold-broadcast.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:freeze_variables.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:func-attr-invalid.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:func-attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-cfg.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:functional-control-flow-to-regions.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:functionalize-if-fail.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:functionalize-if.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:fused_kernel_matcher.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:gpu_fusion.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:graph_pruning.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:graph_pruning_preserve_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:group_by_dialect.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:guarantee-all-funcs-one-use.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:hoist_loop_invariant.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:hoist_replicate_invariant_resource_writes.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:host_launch_to_outside_compiled.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_invalid.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:init_text_file_to_import_saved_model.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:inlining.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:isolate-placer.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:launch_outlining.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:launch_to_device_attribute_legacy.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_60.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_gpu_cc_70.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nchw.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_layout_assignment_to_nhwc.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_begin.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_move_transposes_end.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nchw.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:layout_optimization_to_nhwc.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:legalize_hlo.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_arg_control_dep.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:legalize_tfg_with_control_flow.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:localize_var_handles.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:lower_globals_to_ml_program_invalid.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:lower_quantized.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:lower_tf.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:lower_variable_ops_to_ml_program.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:mark_input_output_aliases.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:mark_ops_for_outside_compilation.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:materialize_passthrough_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:merge_control_flow.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:mlprogram.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:name_anonymous_iterators.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:optimize-arg-operand-constraint.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:optimize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:order_by_dialect.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:outside_compiled_to_host_launch.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:parallel_execute_to_islands_legacy.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:prepare_tpu_computation_for_tf_export.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:promote_resources_to_args_functions.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:promote_var_handles_to_args.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:readonly_references_to_resources.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:region-control-flow-to-functional.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:remove_unused_arguments.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:remove_unused_while_results.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:replica_id_to_device_ordinal.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:replicate_invariant_op_hoisting.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:replicate_tensor_list_init_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:replicate_to_island_legacy.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:resource-alias-analysis-test.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:resource-device-inference.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:resource_analyzer.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:resource_inlining.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:resource_op_lifting.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:rewrite_tpu_embedding_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:roundtrip-tf-executor.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:shape_inference.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:side-effect-analysis-test.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests:sink_constant.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:split_into_island_per_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:stack_ops_decomposition.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:strip_noinline.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:strip_saved_module_metadata.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:strip_tf_attributes.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tensor_array_ops_decomposition.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tensor_list_ops_decomposition.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf-executor-to-functional.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf-functional-to-executor.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf-ops.mlir.test PASSED in 2.5s //tensorflow/compiler/mlir/tensorflow/tests:tf-reduce-identity.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_map_and_batch.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_data_fuse_pmap_and_batch.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_index_selector.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_device_ops_invalid.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_invalid.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_location_roundtrip.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_printer.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:tf_executor_ops_side_effect.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tf_optimize.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_deduplicate_bound_input_bindings.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_assets.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_freeze_global_tensors_mutable_tensors.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_initialize_variables_in_session_init_fail.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_lift_variables_invalid_session.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_mark_initialized_variables.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_ops_invalid.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_optimize_global_tensors_interprocedural.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tf_saved_model_remove_vars_in_session_initializer.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tf_side_effect.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tf_trait_folds.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu-annotate-dynamic-shape-inputs.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:tpu-cluster-cleanup-attributes.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu-dynamic-layout-pass.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu-merge-variables-with-execute.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu-multiple-while-body-func.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu-resource-read-for-write.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu-variable-runtime-reformatting.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:tpu_cluster_formation.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_colocate_composite_resource_ops.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests:tpu_device_propagation.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu_host_computation_expansion.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_identity_pruning.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:tpu_parallel_execute_sink_resource_write.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:tpu_partitioned_op_conversion.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_reorder_replicate_and_partitioned_inputs.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_resource_partitioning.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:tpu_rewrite.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests:tpu_sharding_identification.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests:tpu_space_to_depth_pass.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_tail_with_tobool_op.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_update_embedding_enqueue_op_inputs.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:tpu_validate_inputs.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:transpose-op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests:unroll-batch-matmul.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests:update_control_dependencies.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests:warn_when_using_deprecated_dumps.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:while_licm.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests:xla_cluster_formation.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests:xla_inline_device_ops.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests:xla_rewrite.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:add.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding-invalid.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:argument-sharding.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding-hook.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:constant-folding.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph-resource.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:graph.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:mlir-module-serialized-str-attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:replicate-tensor-list-init-ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:result-sharding.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr-invalid.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:serialized-mlir-module-str-attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference-after-legalization.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:shape-inference.mlir.test PASSED in 1.2s //tensorflow/compiler/mlir/tensorflow/tests/compile_mlir_util:stablehlo_add.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:executor_tpuv1_island_coarsening.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_coarsening:while_op.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:executor_tpuv1_inline_tpu_island.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_island_inlining:while_op.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:case_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:executor_tpuv1_outline_tpu_island.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/executor_tpuv1_outline_island:while_op.mlir.test PASSED in 1.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:add.pbtxt.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-as-fetch.pbtxt.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-control-dep.pbtxt.test PASSED in 1.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type-with-subtype.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-data-type.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-multi-data-type-with-subtype.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:arg-retval-attrs.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:case_op.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:const-values.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:device-arg-retval-attr.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-input-shapes.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:empty-value-attr.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-as-fetch.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:feed-control-dep.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:force_shared_name_for_resource_ops.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:function-func-attr.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-if-ops.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:functional-while-ops.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-control-ret.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function-retval-of-arg.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-as-function.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-custom-operation.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-default-attr.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-device-retval.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-empty-tensor-content.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-func-attr.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-call.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-diff-island.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-control-ret-same-island.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-defs.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-input-shapes.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-name-bug.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-function-resource-args.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-gradient-def.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-input-func-arg-name-collision.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-library.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-malformed.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-scalar-input.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-uint8-return.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-undefined-output.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-version-info.pbtxt.test PASSED in 1.5s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:graph-while-loop.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:invalid-output-index.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:legacy-fed-input-without-inputs.pbtxt.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:merge_node_with_function.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:mlir_passthrough_op.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multi-output-feeds.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:multiple-use-next-iteration.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:node-locations.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes-attr.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:output-shapes.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:parse_example_v2.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:partial-device-name.pbtxt.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:prune_unused_nodes.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:quint8-const.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:shape-attrs.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:stateful-attribute.pbtxt.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:string-attr.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:switch_n.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:target.pbtxt.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tensor-list.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:tf-data-pipeline.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir:unregistered_kernel.pbtxt.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir/batch_use_same_function:saved_model.pbtxt.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:aliasing_arg_attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:case.mlir.test PASSED in 1.4s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:convert_tensor.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_shape_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:derived_size_attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:device-arg-retval-attr.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:export_main_to_flib.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:fetch_feed_names.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:func_list_attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-control-ret.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-order.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args-handle-info.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:function-resource-args.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-if-ops.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:functional-while-ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:graph-as-function.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:infer_derived_attribute.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:invalid_input.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:legalized_name.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:missing-main.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:noop.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:optional_symbol_ref.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:output-shapes-attr.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:parse_example_v2.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:preserve-entry-func-names.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-type-attr.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:ref-while-loop.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:shape_list_attr.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:simple_tf_dialect_op.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:stringescape.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:switchn.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-gradient-attr.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf-legacy-call.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_add.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_identity_n.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:tf_tpu_embedding_ops.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_attr.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:type_list_attr.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_name.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:unique_output_name.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef:while-loop.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tensorflow/tests/tf_to_hlo_pipeline:sccp-post-shape-inference.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tensorflow/tests/tpu_bridge_v1:end_to_end.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tf2xla/api/v0:compile_mlir_util_test PASSED in 5.0s //tensorflow/compiler/mlir/tf2xla/api/v0:compile_tf_graph_test PASSED in 0.3s //tensorflow/compiler/mlir/tf2xla/api/v1:legalize_tf_test PASSED in 17.5s //tensorflow/compiler/mlir/tf2xla/tests:adjust-layout.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/tests:convert-mhlo-quant-to-int.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_runtime_pipeline.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tf2xla/tests:hlo_xla_sparsification.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-BatchMatMulV2.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-binary-elementwise.mlir.test PASSED in 1.6s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-collective.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-communication.mlir.test PASSED in 2.0s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-include-tf2xla-fallback.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-no-tf2xla-fallback.mlir.test PASSED in 4.5s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-prefer-tf2xla.mlir.test PASSED in 1.7s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-types.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-with-tf2xla-hlo-importer-and-inline.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-with-tf2xla-hlo-importer.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf-with-tf2xla.mlir.test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/tests:legalize-tf.mlir.test PASSED in 14.5s //tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_cpu.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tf2xla/tests:tfxla_device_specific_transformations_gpu.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization-no-chlo.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/tests:verify-tfxla-legalization.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tf2xla/transforms:tf2xla_rewriter_test PASSED in 14.6s //tensorflow/compiler/mlir/tf2xla/transforms:verify_tfxla_legalization_test PASSED in 11.6s //tensorflow/compiler/mlir/tf2xla/transforms:xla_legalize_targets_test PASSED in 1.0s //tensorflow/compiler/mlir/tf2xla/transforms:xla_legalize_tf_test PASSED in 3.4s //tensorflow/compiler/mlir/tfr:graph_decompose_test PASSED in 18.6s //tensorflow/compiler/mlir/tfr:node_expansion_test PASSED in 16.6s //tensorflow/compiler/mlir/tfr:op_reg_gen_test PASSED in 70.8s //tensorflow/compiler/mlir/tfr:tfr_decompose_ctx_test PASSED in 5.6s //tensorflow/compiler/mlir/tfr:tfr_gen_test PASSED in 15.9s //tensorflow/compiler/mlir/tfr/examples/customization:test_ops_test PASSED in 13.2s //tensorflow/compiler/mlir/tfr/examples/pad:pad_ops_test PASSED in 17.5s //tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_deallocation.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tools/kernel_gen/tests:buffer_reuse.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tools/kernel_gen/tests:bufferize.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:copy_cleanup.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:embed_tf_framework.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:invalid.mlir.test PASSED in 1.1s //tensorflow/compiler/mlir/tools/kernel_gen/tests:isinf.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:ops.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:parallel_loops_to_sequential.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tools/kernel_gen/tests:rewrite_tf_framework_assert.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tanh.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf-legalize-to-lmhlo.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_abi_knowledge.mlir.test PASSED in 0.9s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_framework_legalize_to_llvm.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_kernel_gpu_launch_to_llvm.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tools/kernel_gen/tests:tf_to_jit_invocations.mlir.test PASSED in 1.5s //tensorflow/compiler/mlir/tosa/tests:convert-tfl-uint8.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:convert_metadata.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:fuse-bias-tf.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tosa/tests:lower-complex-types.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:lower_global_tensors.mlir.test PASSED in 0.5s //tensorflow/compiler/mlir/tosa/tests:multi_add.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:retain_call_once_funcs.mlir.test PASSED in 1.8s //tensorflow/compiler/mlir/tosa/tests:strip-quant-types.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:strip_metadata.mlir.test PASSED in 0.6s //tensorflow/compiler/mlir/tosa/tests:tf-tfl-to-tosa-pipeline.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:tf-to-tosa-pipeline.mlir.test PASSED in 1.3s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-dequantize_softmax.mlir.test PASSED in 0.7s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline-filtered.mlir.test PASSED in 0.8s //tensorflow/compiler/mlir/tosa/tests:tfl-to-tosa-pipeline.mlir.test PASSED in 6.3s //tensorflow/compiler/mlir/tosa/tests:verify_fully_converted.mlir.test PASSED in 0.5s //tensorflow/compiler/tests:adadelta_test_cpu PASSED in 11.8s //tensorflow/compiler/tests:adagrad_da_test_cpu PASSED in 9.2s //tensorflow/compiler/tests:adagrad_test_cpu PASSED in 8.2s //tensorflow/compiler/tests:adam_test_cpu PASSED in 11.0s //tensorflow/compiler/tests:add_n_test_cpu PASSED in 6.8s //tensorflow/compiler/tests:argminmax_test_cpu PASSED in 34.5s //tensorflow/compiler/tests:argminmax_test_cpu_mlir_bridge_test PASSED in 11.7s //tensorflow/compiler/tests:bucketize_op_test_cpu PASSED in 6.6s //tensorflow/compiler/tests:bucketize_op_test_cpu_mlir_bridge_test PASSED in 6.4s //tensorflow/compiler/tests:case_test_cpu PASSED in 6.1s //tensorflow/compiler/tests:cast_ops_test_cpu PASSED in 6.3s //tensorflow/compiler/tests:cast_ops_test_cpu_mlir_bridge_test PASSED in 6.1s //tensorflow/compiler/tests:categorical_op_test_cpu PASSED in 9.5s //tensorflow/compiler/tests:categorical_op_test_cpu_mlir_bridge_test PASSED in 10.1s //tensorflow/compiler/tests:cholesky_op_test_cpu PASSED in 15.7s //tensorflow/compiler/tests:cholesky_op_test_cpu_mlir_bridge_test PASSED in 15.4s //tensorflow/compiler/tests:clustering_test_cpu PASSED in 6.2s //tensorflow/compiler/tests:clustering_test_cpu_mlir_bridge_test PASSED in 6.8s //tensorflow/compiler/tests:concat_ops_test_cpu PASSED in 8.6s //tensorflow/compiler/tests:concat_ops_test_cpu_mlir_bridge_test PASSED in 10.6s //tensorflow/compiler/tests:cond_test_cpu PASSED in 6.6s //tensorflow/compiler/tests:const_arg_test_cpu PASSED in 6.0s //tensorflow/compiler/tests:const_test_cpu PASSED in 6.7s //tensorflow/compiler/tests:data_format_ops_test_cpu PASSED in 9.9s //tensorflow/compiler/tests:data_format_ops_test_cpu_mlir_bridge_test PASSED in 12.6s //tensorflow/compiler/tests:dense_layer_test_cpu PASSED in 9.6s //tensorflow/compiler/tests:dynamic_slice_ops_test_cpu PASSED in 9.7s //tensorflow/compiler/tests:dynamic_slice_ops_test_cpu_mlir_bridge_test PASSED in 9.7s //tensorflow/compiler/tests:dynamic_stitch_test_cpu PASSED in 6.4s //tensorflow/compiler/tests:dynamic_stitch_test_cpu_mlir_bridge_test PASSED in 6.9s //tensorflow/compiler/tests:eager_test_cpu PASSED in 14.9s //tensorflow/compiler/tests:einsum_op_test_cpu PASSED in 7.4s //tensorflow/compiler/tests:einsum_op_test_cpu_mlir_bridge_test PASSED in 7.4s //tensorflow/compiler/tests:ensure_shape_op_test_cpu PASSED in 6.6s //tensorflow/compiler/tests:extract_image_patches_op_test_cpu PASSED in 7.1s //tensorflow/compiler/tests:extract_image_patches_op_test_cpu_mlir_bridge_test PASSED in 6.1s //tensorflow/compiler/tests:fake_quant_ops_test_cpu PASSED in 11.7s //tensorflow/compiler/tests:fake_quant_ops_test_cpu_mlir_bridge_test PASSED in 14.8s //tensorflow/compiler/tests:fifo_queue_test_cpu PASSED in 23.2s //tensorflow/compiler/tests:fifo_queue_test_cpu_mlir_bridge_test PASSED in 8.3s //tensorflow/compiler/tests:ftrl_ops_test_cpu PASSED in 13.0s //tensorflow/compiler/tests:ftrl_ops_test_cpu_mlir_bridge_test PASSED in 16.4s //tensorflow/compiler/tests:ftrl_test_cpu PASSED in 13.9s //tensorflow/compiler/tests:function_test_cpu PASSED in 5.9s //tensorflow/compiler/tests:function_test_cpu_mlir_bridge_test PASSED in 6.4s //tensorflow/compiler/tests:gather_nd_op_test_cpu PASSED in 7.3s //tensorflow/compiler/tests:gather_nd_op_test_cpu_mlir_bridge_test PASSED in 7.6s //tensorflow/compiler/tests:gather_test_cpu PASSED in 66.9s //tensorflow/compiler/tests:gather_test_cpu_mlir_bridge_test PASSED in 51.0s //tensorflow/compiler/tests:jit_test_cpu PASSED in 45.0s //tensorflow/compiler/tests:listdiff_op_test_cpu PASSED in 9.0s //tensorflow/compiler/tests:listdiff_op_test_cpu_mlir_bridge_test PASSED in 11.5s //tensorflow/compiler/tests:lrn_ops_test_cpu PASSED in 6.5s //tensorflow/compiler/tests:lrn_ops_test_cpu_mlir_bridge_test PASSED in 7.4s //tensorflow/compiler/tests:lstm_test_cpu PASSED in 20.6s //tensorflow/compiler/tests:manip_ops_test_cpu PASSED in 9.2s //tensorflow/compiler/tests:manip_ops_test_cpu_mlir_bridge_test PASSED in 10.8s //tensorflow/compiler/tests:matrix_band_part_test_cpu PASSED in 37.4s //tensorflow/compiler/tests:matrix_band_part_test_cpu_mlir_bridge_test PASSED in 35.1s //tensorflow/compiler/tests:matrix_inverse_op_test_cpu PASSED in 16.0s //tensorflow/compiler/tests:matrix_inverse_op_test_cpu_mlir_bridge_test PASSED in 19.0s //tensorflow/compiler/tests:matrix_solve_op_test_cpu PASSED in 8.0s //tensorflow/compiler/tests:matrix_solve_op_test_cpu_mlir_bridge_test PASSED in 7.0s //tensorflow/compiler/tests:matrix_triangular_solve_op_test_cpu PASSED in 22.3s //tensorflow/compiler/tests:matrix_triangular_solve_op_test_cpu_mlir_bridge_test PASSED in 26.6s //tensorflow/compiler/tests:momentum_test_cpu PASSED in 8.7s //tensorflow/compiler/tests:nary_ops_test_cpu PASSED in 7.4s //tensorflow/compiler/tests:nary_ops_test_cpu_mlir_bridge_test PASSED in 8.9s //tensorflow/compiler/tests:nullary_ops_test_cpu PASSED in 7.4s //tensorflow/compiler/tests:nullary_ops_test_cpu_mlir_bridge_test PASSED in 6.9s //tensorflow/compiler/tests:placeholder_test_cpu PASSED in 26.0s //tensorflow/compiler/tests:placeholder_test_cpu_mlir_bridge_test PASSED in 6.3s //tensorflow/compiler/tests:proximal_adagrad_test_cpu PASSED in 6.8s //tensorflow/compiler/tests:proximal_gradient_descent_test_cpu PASSED in 6.2s //tensorflow/compiler/tests:quantized_ops_test_cpu PASSED in 6.6s //tensorflow/compiler/tests:reduce_window_test_cpu PASSED in 6.6s //tensorflow/compiler/tests:reduce_window_test_cpu_mlir_bridge_test PASSED in 7.1s //tensorflow/compiler/tests:reshape_op_test_cpu PASSED in 7.6s //tensorflow/compiler/tests:reshape_op_test_cpu_mlir_bridge_test PASSED in 8.0s //tensorflow/compiler/tests:reverse_ops_test_cpu PASSED in 9.8s //tensorflow/compiler/tests:reverse_ops_test_cpu_mlir_bridge_test PASSED in 26.5s //tensorflow/compiler/tests:reverse_sequence_op_test_cpu PASSED in 8.1s //tensorflow/compiler/tests:reverse_sequence_op_test_cpu_mlir_bridge_test PASSED in 7.7s //tensorflow/compiler/tests:risc_ops_test_cpu_mlir_bridge_test PASSED in 6.3s //tensorflow/compiler/tests:rmsprop_test_cpu PASSED in 35.0s //tensorflow/compiler/tests:scatter_nd_op_test_cpu PASSED in 18.4s //tensorflow/compiler/tests:scatter_nd_op_test_cpu_mlir_bridge_test PASSED in 31.1s //tensorflow/compiler/tests:searchsorted_op_test_cpu PASSED in 8.4s //tensorflow/compiler/tests:searchsorted_op_test_cpu_mlir_bridge_test PASSED in 8.5s //tensorflow/compiler/tests:segment_reduction_ops_test_cpu PASSED in 20.9s //tensorflow/compiler/tests:segment_reduction_ops_test_cpu_mlir_bridge_test PASSED in 28.2s //tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu PASSED in 14.9s //tensorflow/compiler/tests:self_adjoint_eig_op_test_cpu_mlir_bridge_test PASSED in 15.2s //tensorflow/compiler/tests:slice_ops_test_cpu PASSED in 16.1s //tensorflow/compiler/tests:slice_ops_test_cpu_mlir_bridge_test PASSED in 17.3s //tensorflow/compiler/tests:sparse_to_dense_op_test_cpu PASSED in 7.8s //tensorflow/compiler/tests:sparse_to_dense_op_test_cpu_mlir_bridge_test PASSED in 22.2s //tensorflow/compiler/tests:stack_ops_test_cpu PASSED in 6.8s //tensorflow/compiler/tests:tensor_list_ops_test_cpu PASSED in 7.3s //tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu PASSED in 12.0s //tensorflow/compiler/tests:tridiagonal_matmul_ops_test_cpu_mlir_bridge_test PASSED in 37.1s //tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu PASSED in 10.3s //tensorflow/compiler/tests:tridiagonal_solve_ops_test_cpu_mlir_bridge_test PASSED in 38.0s //tensorflow/compiler/tests:unique_ops_test_cpu PASSED in 6.8s //tensorflow/compiler/tests:variable_ops_test_cpu PASSED in 23.2s //tensorflow/compiler/tests:variable_ops_test_cpu_mlir_bridge_test PASSED in 12.3s //tensorflow/compiler/tests:where_op_test_cpu PASSED in 13.6s //tensorflow/compiler/tests:while_test_cpu PASSED in 31.4s //tensorflow/compiler/tests:xla_call_module_test_cpu PASSED in 7.7s //tensorflow/compiler/tests:xla_custom_call_ops_test_cpu PASSED in 33.4s //tensorflow/compiler/tests:xla_device_gpu_test_cpu PASSED in 21.3s //tensorflow/compiler/tests:xla_device_test_cpu PASSED in 13.2s //tensorflow/compiler/tests:xla_device_test_cpu_mlir_bridge_test PASSED in 14.9s //tensorflow/compiler/tests:xla_ops_test_cpu PASSED in 35.5s //tensorflow/compiler/tests:xla_ops_test_cpu_mlir_bridge_test PASSED in 38.0s //tensorflow/compiler/tests:xla_test_test PASSED in 6.3s //tensorflow/compiler/tf2xla:const_analysis_test PASSED in 5.5s //tensorflow/compiler/tf2xla:cpu_function_runtime_test PASSED in 0.4s //tensorflow/compiler/tf2xla:functionalize_cond_test PASSED in 0.7s //tensorflow/compiler/tf2xla:functionalize_control_flow_test PASSED in 0.7s //tensorflow/compiler/tf2xla:fused_batchnorm_reserve_space_test_cpu PASSED in 27.0s //tensorflow/compiler/tf2xla:graph_compiler_test PASSED in 4.7s //tensorflow/compiler/tf2xla:literal_util_test PASSED in 1.2s //tensorflow/compiler/tf2xla:resource_operation_table_test PASSED in 8.0s //tensorflow/compiler/tf2xla:resource_util_test_cpu PASSED in 2.4s //tensorflow/compiler/tf2xla:sharding_util_test PASSED in 0.5s //tensorflow/compiler/tf2xla:tf2xla_opset_test PASSED in 8.9s //tensorflow/compiler/tf2xla:tf2xla_test PASSED in 16.5s //tensorflow/compiler/tf2xla:tf2xla_util_test PASSED in 0.7s //tensorflow/compiler/tf2xla:xla_compiler_test PASSED in 13.7s //tensorflow/compiler/tf2xla:xla_jit_compiled_cpu_function_test PASSED in 15.9s //tensorflow/compiler/tf2xla:xla_op_registry_test PASSED in 4.6s //tensorflow/compiler/tf2xla/kernels:rng_converter_utils_test PASSED in 1.4s //tensorflow/compiler/xla:array2d_test PASSED in 0.1s //tensorflow/compiler/xla:array3d_test PASSED in 0.2s //tensorflow/compiler/xla:array4d_test PASSED in 0.6s //tensorflow/compiler/xla:array_test PASSED in 0.3s //tensorflow/compiler/xla:bit_cast_test PASSED in 0.1s //tensorflow/compiler/xla:comparison_util_test PASSED in 0.2s //tensorflow/compiler/xla:debug_options_parsers_test PASSED in 0.3s //tensorflow/compiler/xla:index_util_test PASSED in 0.2s //tensorflow/compiler/xla:iterator_util_test PASSED in 0.2s //tensorflow/compiler/xla:layout_test PASSED in 0.2s //tensorflow/compiler/xla:layout_util_test PASSED in 0.3s //tensorflow/compiler/xla:literal_test PASSED in 0.2s //tensorflow/compiler/xla:parse_flags_from_env_test PASSED in 0.9s //tensorflow/compiler/xla:permutation_util_test PASSED in 0.2s //tensorflow/compiler/xla:primitive_util_test PASSED in 0.2s //tensorflow/compiler/xla:refcounting_hash_map_test PASSED in 0.2s //tensorflow/compiler/xla:reference_util_test PASSED in 0.3s //tensorflow/compiler/xla:shape_test PASSED in 0.2s //tensorflow/compiler/xla:shape_tree_test PASSED in 0.1s //tensorflow/compiler/xla:shape_util_test PASSED in 2.5s //tensorflow/compiler/xla:status_macros_test PASSED in 0.1s //tensorflow/compiler/xla:text_literal_reader_test PASSED in 0.2s //tensorflow/compiler/xla:text_literal_writer_test PASSED in 0.2s //tensorflow/compiler/xla:types_test PASSED in 0.3s //tensorflow/compiler/xla:util_test PASSED in 0.2s //tensorflow/compiler/xla:window_util_test PASSED in 0.2s //tensorflow/compiler/xla/client:padding_test PASSED in 0.1s //tensorflow/compiler/xla/client:xla_builder_test PASSED in 0.3s //tensorflow/compiler/xla/client/lib:arithmetic_test_cpu PASSED in 8.5s //tensorflow/compiler/xla/client/lib:comparators_test_cpu PASSED in 7.6s //tensorflow/compiler/xla/client/lib:constants_test_cpu PASSED in 7.5s //tensorflow/compiler/xla/client/lib:logdet_test_cpu PASSED in 13.0s //tensorflow/compiler/xla/client/lib:math_test_cpu PASSED in 14.3s //tensorflow/compiler/xla/client/lib:matrix_test_cpu PASSED in 11.9s //tensorflow/compiler/xla/client/lib:pooling_test_cpu PASSED in 8.6s //tensorflow/compiler/xla/client/lib:qr_test_cpu PASSED in 11.9s //tensorflow/compiler/xla/client/lib:slicing_test_cpu PASSED in 8.5s //tensorflow/compiler/xla/client/lib:sorting_test_cpu PASSED in 8.4s //tensorflow/compiler/xla/examples/axpy:stablehlo_compile_test PASSED in 10.2s //tensorflow/compiler/xla/experiments/sm_bandwidth_benchmark:sm_bw_test PASSED in 0.0s //tensorflow/compiler/xla/hlo/evaluator:hlo_evaluator_test PASSED in 4.1s //tensorflow/compiler/xla/hlo/experimental/auto_sharding:auto_sharding_test PASSED in 2.1s //tensorflow/compiler/xla/hlo/transforms:hlo_constant_splitter_test PASSED in 0.9s //tensorflow/compiler/xla/hlo/utils:hlo_live_range_test PASSED in 1.3s //tensorflow/compiler/xla/hlo/utils:hlo_matchers_test PASSED in 1.1s //tensorflow/compiler/xla/hlo/utils:hlo_sharding_util_test PASSED in 0.2s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:collective_ops.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:collective_ops_to_cpu_runtime.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:fft.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:legalize_i1_vector_transfers.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:lmhlo_custom_call.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:lmhlo_infeed.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:remove_copies_to_out_params.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:rng_bit_generator.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_abi_legalization.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_memref_element_cast_to_llvm.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/cpu/transforms/tests:xla_cpu_outfeed.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:add_hlo_trace.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_launch.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memcpy.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:gpu_memset.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_case.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_custom_call.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_fft.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cholesky.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_conv.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_cublas_lt_matmul.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_gpu_gemm.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_infeed.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_outfeed.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_send_recv.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:lmhlo_while.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:memref_get_global_to_arg.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/backends/gpu/transforms/tests:outline_cuda_graphs.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/framework/tests:legalize-xla-framework.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/framework/tests:outline-with-xla-framework.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/framework/tests:xla-framework.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/math/transforms/tests:math_optimization.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/memref/transforms/tests:aligned_allocations.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/runtime/ir/tests:ops.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/runtime/ir/tests:ops_verify.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/runtime/ir/tests:testlib.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/runtime/transforms:calling_convention_test PASSED in 0.4s //tensorflow/compiler/xla/mlir/runtime/transforms:type_converter_test PASSED in 0.2s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:compilation_pipeline.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_asserts.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:convert_custom_calls.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:export_functions.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:ordinal_assignment.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/runtime/transforms/tests:rt_to_llvm.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:erase-op-without-results.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:inline-scf-while.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:reduce-scf-forall-bounds.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-constant.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-op-with-value.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:replace-operand-with-constant.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:return-operands-of-terminator-operands.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/rewrites/tests:truncate-function.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:bisect.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:no-bug.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/tools/mlir_bisect/tests:snapshot.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/tools/mlir_replay/public:execution_trace_utils_test PASSED in 0.2s //tensorflow/compiler/xla/mlir/utils:error_util_test PASSED in 0.1s //tensorflow/compiler/xla/mlir/xla_cpu/tests:bufferize.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir/xla_cpu/tests:invalid.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir/xla_cpu/tests:ops.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/bufferization/hlo_one_shot_bufferize.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_hlo_broadcasts.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_hlo_no_broadcasts.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/chlo_legalize_to_mhlo.mlir.test PASSED in 1.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/chlo/sparse_chlo_legalize_to_linalg.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/buffer_reuse.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/convert_deallocation_ops_to_llvm.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocate.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocate_invalid.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_ops.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_simplification.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/deallocation_to_scf.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/deallocation/split_alloc_tensors.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/add_debug_info.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/bufferization.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/collapse-shape.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/collect_stats.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/compose_extract_insert_slice.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/batch_matmul.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/conv_2d_nhwc_hwcf.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/dot.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/duplicate_fusions.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fibonacci.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fusion_outlining.mlir.test PASSED in 1.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/fusion_planning_for_cpu.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/inline_fusion_clusters.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_bcast_map.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_matmul.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_reduce.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_reduce_map.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/map_reshape_map.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/matmul.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_1d.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_1d_map.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_2d.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reduce_window.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/reverse.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/scatter.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/sort.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/cpu_tiling/transpose.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/greedy_fusion.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/invalid.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/lower_vectors.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/nested_tiling_softmax.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/ops.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/optimize_linalg_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/rewrite_forall_to_for.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/simplify_dead_copy.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/tile_by_one.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/tiling_softmax.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/vectorize_copy.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/gml_st/vectorize_for_cpu.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-select-and-scatter.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-affine.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-gpu.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-parallel-loops.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/lhlo-legalize-to-tensor-op.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo/ops.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/lhlo_gpu/lhlo_gpu_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/attrs.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/broadcast_propagation.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/bitcast.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/canonicalize.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/concatenate.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/convert.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/convolution.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/custom_call.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/folder_limit.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reduce.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reshape.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/reverse.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/scatter.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/transpose.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/tuple.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/canonicalize/while.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/constraint_fusion.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/convert_to_signless.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/expand_hlo_tuples.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/expand_ops_simplifier.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/group_reduction_dimensions.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-collapse-elementwise-map.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-einsum-to-dot-general.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-gather-to-torch-index-select.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-rng-to-linalg.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-shape-ops-to-standard.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-sort.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-arithmetic.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo-only-dynamic.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo-unranked.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-lhlo.mlir.test PASSED in 1.0s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-linalg.mlir.test PASSED in 3.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-memref-unranked.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-memref.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-stablehlo-experimental.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/hlo-legalize-to-stablehlo.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/inlining.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/invalid.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-control-flow.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-hlo-shape-computations.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-mhlo-to-thlo.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/legalize-to-std.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/lower-complex.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/lower-general-dot.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/materialize-broadcasts.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/merge_assuming_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_bytecode_customizations.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_dot.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_gather.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_reduction.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_canonicalize_scatter.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_flatten_tuple.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_infer_shape_type_methods.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_ops_prettyprint.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/mhlo_reduce_pretty_print.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/ops.mlir.test PASSED in 1.2s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/optimize-hlo.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/prepare-for-export.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/reify-result-types.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/restrict_max_rank.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/shape_legalize_to_hlo.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/shape_reification.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sink-constants-to-control-flow.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_gendot_lower.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_lower.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_ops.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_rewriting.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/sparse_transpose.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/stablehlo-legalize-to-hlo.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/symbolic-shape-optimization.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/unfuse_batch_norm.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_bounds.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_conv_op.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_reduce_op.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_reduce_window_op.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_scatter_op.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_select_and_scatter_op.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/verifier_while_op.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/mhlo/while_prettyprint.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/bufferize.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/canonicalize.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/invalid.mlir.test PASSED in 1.3s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/legalize_sort.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/ops.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:Dialect/thlo/tiling.mlir.test PASSED in 0.9s //tensorflow/compiler/xla/mlir_hlo/tests:alloc_to_arg.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:assuming-structural-propagation.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:buffer_packing.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:bufferize.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:bufferize_one_shot.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:collapse_parallel_loops_to_1d_pass.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:detensorize_scf_ops.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:index_type_llvm_lowering.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:legalize-trigonometric-to-approximation.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:lower_index_cast.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:propagate_static_shapes.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:rank-specialization.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:scalarization.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:shape-component-analysis.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tests:shape_simplification.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tests:test_userange.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tests:tile_loops.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/mlir_hlo/tests:unbufferize.mlir.test PASSED in 1.9s //tensorflow/compiler/xla/mlir_hlo/tests:unroll-loops.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tools/mlir_interpreter/framework/tests:interpreter_value_test PASSED in 0.5s //tensorflow/compiler/xla/mlir_hlo/tools/mlir_interpreter/framework/tests:tensor_or_memref_test PASSED in 0.2s //tensorflow/compiler/xla/mlir_hlo/tosa/tests:binary.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tosa/tests:nullary.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/mlir_hlo/tosa/tests:prepare-mhlo.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tosa/tests:ternary.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/mlir_hlo/tosa/tests:unary.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/pjrt:host_callback_test PASSED in 0.3s //tensorflow/compiler/xla/pjrt:lru_cache_test PASSED in 0.3s //tensorflow/compiler/xla/pjrt:pjrt_api_test PASSED in 0.2s //tensorflow/compiler/xla/pjrt:pjrt_client_test_cpu PASSED in 7.8s //tensorflow/compiler/xla/pjrt:pjrt_compiler_test PASSED in 0.2s //tensorflow/compiler/xla/pjrt:pjrt_executable_test PASSED in 0.2s //tensorflow/compiler/xla/pjrt:pjrt_stream_executor_client_test PASSED in 7.9s //tensorflow/compiler/xla/pjrt:semaphore_test PASSED in 0.2s //tensorflow/compiler/xla/pjrt:tf_pjrt_client_test PASSED in 5.6s //tensorflow/compiler/xla/pjrt:tfrt_cpu_pjrt_client_test PASSED in 7.2s //tensorflow/compiler/xla/pjrt:tracked_device_buffer_test PASSED in 7.3s //tensorflow/compiler/xla/pjrt:tracked_tfrt_cpu_device_buffer_test PASSED in 0.5s //tensorflow/compiler/xla/pjrt:transpose_test PASSED in 40.8s //tensorflow/compiler/xla/pjrt/c:pjrt_c_api_cpu_test PASSED in 7.1s //tensorflow/compiler/xla/pjrt/c:pjrt_c_api_helpers_test PASSED in 0.5s //tensorflow/compiler/xla/pjrt/distributed:client_server_test PASSED in 41.6s //tensorflow/compiler/xla/pjrt/distributed:service_test PASSED in 6.7s //tensorflow/compiler/xla/pjrt/gpu:se_gpu_pjrt_client_test PASSED in 1.8s //tensorflow/compiler/xla/python:outfeed_receiver_test_cpu PASSED in 8.2s //tensorflow/compiler/xla/python/ifrt:array_test PASSED in 0.2s //tensorflow/compiler/xla/python/ifrt:array_test_no_impl PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:client_test_no_impl PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:executable_test_no_impl PASSED in 1.5s //tensorflow/compiler/xla/python/ifrt:future_test PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:index_domain_test PASSED in 0.3s //tensorflow/compiler/xla/python/ifrt:index_test PASSED in 0.4s //tensorflow/compiler/xla/python/ifrt:shape_test PASSED in 0.2s //tensorflow/compiler/xla/python/ifrt:sharding_test PASSED in 0.2s //tensorflow/compiler/xla/python/ifrt:tuple_test_no_impl PASSED in 0.2s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_array.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_assemble.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_call.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_call_loaded_executable.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_disassemble.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_loaded_executable.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/python/ifrt/ir/tests:verify_reshard.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/python/ifrt/support:sharding_param_to_op_sharding_test PASSED in 0.2s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_array_impl_test_tfrt_cpu PASSED in 13.0s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_client_impl_test_tfrt_cpu PASSED in 6.5s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_executable_impl_test_tfrt_cpu PASSED in 5.8s //tensorflow/compiler/xla/python/pjrt_ifrt:pjrt_tuple_impl_test_tfrt_cpu PASSED in 6.2s //tensorflow/compiler/xla/python_api:xla_literal_test PASSED in 0.8s //tensorflow/compiler/xla/python_api:xla_shape_test PASSED in 0.7s //tensorflow/compiler/xla/rpc:grpc_client_test PASSED in 2.6s //tensorflow/compiler/xla/runtime:arguments_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:async_runtime_test PASSED in 0.3s //tensorflow/compiler/xla/runtime:custom_call_test PASSED in 1.5s //tensorflow/compiler/xla/runtime:diagnostics_test PASSED in 0.1s //tensorflow/compiler/xla/runtime:executable_test PASSED in 1.8s //tensorflow/compiler/xla/runtime:ffi_test PASSED in 1.0s //tensorflow/compiler/xla/runtime:map_by_type_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:module_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:results_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:state_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:symbolic_shape_test PASSED in 0.2s //tensorflow/compiler/xla/runtime:type_id_test PASSED in 0.1s //tensorflow/compiler/xla/service:algebraic_simplifier_overflow_test_cpu PASSED in 6.8s //tensorflow/compiler/xla/service:algebraic_simplifier_test PASSED in 2.3s //tensorflow/compiler/xla/service:all_gather_broadcast_reorder_test PASSED in 1.1s //tensorflow/compiler/xla/service:all_gather_combiner_test PASSED in 1.6s //tensorflow/compiler/xla/service:all_gather_decomposer_test PASSED in 0.9s //tensorflow/compiler/xla/service:all_reduce_combiner_test PASSED in 1.0s //tensorflow/compiler/xla/service:all_reduce_contiguous_test PASSED in 0.8s //tensorflow/compiler/xla/service:all_reduce_folder_test PASSED in 1.4s //tensorflow/compiler/xla/service:all_reduce_promotion_test PASSED in 1.5s //tensorflow/compiler/xla/service:all_reduce_reassociate_test PASSED in 0.9s //tensorflow/compiler/xla/service:all_reduce_simplifier_test PASSED in 1.1s //tensorflow/compiler/xla/service:ar_crs_combiner_test PASSED in 1.0s //tensorflow/compiler/xla/service:async_collective_creator_test PASSED in 0.9s //tensorflow/compiler/xla/service:async_op_canonicalizer_test PASSED in 1.0s //tensorflow/compiler/xla/service:batch_dot_simplification_test PASSED in 0.8s //tensorflow/compiler/xla/service:batchnorm_expander_test_cpu PASSED in 8.6s //tensorflow/compiler/xla/service:bfloat16_conversion_folding_test PASSED in 1.0s //tensorflow/compiler/xla/service:bfloat16_propagation_test PASSED in 1.1s //tensorflow/compiler/xla/service:bitcast_dtypes_expander_test PASSED in 0.8s //tensorflow/compiler/xla/service:broadcast_canonicalizer_test PASSED in 1.4s //tensorflow/compiler/xla/service:buffer_assignment_test PASSED in 11.2s //tensorflow/compiler/xla/service:call_graph_test PASSED in 1.2s //tensorflow/compiler/xla/service:call_inliner_test PASSED in 1.2s //tensorflow/compiler/xla/service:change_op_data_type_test PASSED in 2.0s //tensorflow/compiler/xla/service:collective_ops_utils_test PASSED in 0.1s //tensorflow/compiler/xla/service:collectives_schedule_linearizer_test PASSED in 0.8s //tensorflow/compiler/xla/service:compilation_environments_test PASSED in 0.2s //tensorflow/compiler/xla/service:conditional_canonicalizer_test PASSED in 1.1s //tensorflow/compiler/xla/service:conditional_code_motion_test PASSED in 0.8s //tensorflow/compiler/xla/service:conditional_simplifier_test PASSED in 1.0s //tensorflow/compiler/xla/service:conditional_to_select_test PASSED in 0.8s //tensorflow/compiler/xla/service:convert_async_collectives_to_sync_test PASSED in 0.8s //tensorflow/compiler/xla/service:convert_mover_test PASSED in 1.0s //tensorflow/compiler/xla/service:convert_operand_folding_test PASSED in 0.9s //tensorflow/compiler/xla/service:convolution_4d_expander_test PASSED in 1.1s //tensorflow/compiler/xla/service:convolution_group_converter_test PASSED in 0.8s //tensorflow/compiler/xla/service:convolution_pred_expander_test PASSED in 0.9s //tensorflow/compiler/xla/service:copy_insertion_test PASSED in 1.1s //tensorflow/compiler/xla/service:custom_call_status_test PASSED in 0.2s //tensorflow/compiler/xla/service:defuser_test PASSED in 0.9s //tensorflow/compiler/xla/service:despecializer_test PASSED in 0.9s //tensorflow/compiler/xla/service:dfs_hlo_visitor_with_default_test PASSED in 0.9s //tensorflow/compiler/xla/service:dot_decomposer_test PASSED in 0.9s //tensorflow/compiler/xla/service:dot_merger_test PASSED in 1.2s //tensorflow/compiler/xla/service:dynamic_dimension_inference_test PASSED in 1.0s //tensorflow/compiler/xla/service:dynamic_dimension_simplifier_test PASSED in 0.9s //tensorflow/compiler/xla/service:dynamic_index_splitter_test PASSED in 0.8s //tensorflow/compiler/xla/service:dynamic_padder_test_cpu PASSED in 11.2s //tensorflow/compiler/xla/service:dynamic_parameter_binding_test PASSED in 0.9s //tensorflow/compiler/xla/service:dynamic_update_slice_test_cpu PASSED in 8.0s //tensorflow/compiler/xla/service:elemental_ir_emitter_test_cpu PASSED in 8.0s //tensorflow/compiler/xla/service:flatten_call_graph_test PASSED in 1.0s //tensorflow/compiler/xla/service:float_normalization_test PASSED in 0.9s //tensorflow/compiler/xla/service:fusion_node_indexing_evaluation_test PASSED in 1.0s //tensorflow/compiler/xla/service:gather_expander_test PASSED in 1.1s //tensorflow/compiler/xla/service:gather_simplifier_test PASSED in 1.0s //tensorflow/compiler/xla/service:heap_simulator_test PASSED in 1.9s //tensorflow/compiler/xla/service:hlo_alias_analysis_test PASSED in 1.1s //tensorflow/compiler/xla/service:hlo_casting_utils_test PASSED in 10.4s //tensorflow/compiler/xla/service:hlo_computation_deduplicator_test PASSED in 2.3s //tensorflow/compiler/xla/service:hlo_computation_test PASSED in 2.4s //tensorflow/compiler/xla/service:hlo_constant_folding_test PASSED in 5.2s //tensorflow/compiler/xla/service:hlo_cost_analysis_test PASSED in 6.5s //tensorflow/compiler/xla/service:hlo_creation_utils_test PASSED in 3.9s //tensorflow/compiler/xla/service:hlo_cse_test PASSED in 10.0s //tensorflow/compiler/xla/service:hlo_dataflow_analysis_test PASSED in 1.2s //tensorflow/compiler/xla/service:hlo_dce_test PASSED in 1.3s //tensorflow/compiler/xla/service:hlo_domain_test PASSED in 0.9s //tensorflow/compiler/xla/service:hlo_element_type_converter_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_execution_profile_test PASSED in 5.9s //tensorflow/compiler/xla/service:hlo_graph_dumper_test PASSED in 0.9s //tensorflow/compiler/xla/service:hlo_input_output_alias_config_test PASSED in 0.9s //tensorflow/compiler/xla/service:hlo_instruction_test PASSED in 1.0s //tensorflow/compiler/xla/service:hlo_liveness_analysis_test PASSED in 0.9s //tensorflow/compiler/xla/service:hlo_memory_scheduler_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_module_dce_test PASSED in 1.3s //tensorflow/compiler/xla/service:hlo_module_metadata_test PASSED in 0.2s //tensorflow/compiler/xla/service:hlo_module_test PASSED in 1.9s //tensorflow/compiler/xla/service:hlo_opcode_test PASSED in 0.3s //tensorflow/compiler/xla/service:hlo_ordering_test PASSED in 1.0s //tensorflow/compiler/xla/service:hlo_parser_test PASSED in 0.4s //tensorflow/compiler/xla/service:hlo_pass_pipeline_test PASSED in 1.1s //tensorflow/compiler/xla/service:hlo_phi_graph_test PASSED in 0.3s //tensorflow/compiler/xla/service:hlo_proto_util_test PASSED in 0.9s //tensorflow/compiler/xla/service:hlo_reachability_test PASSED in 1.2s //tensorflow/compiler/xla/service:hlo_rematerialization_test PASSED in 1.2s //tensorflow/compiler/xla/service:hlo_rematerialization_test_utils_test PASSED in 1.2s //tensorflow/compiler/xla/service:hlo_replication_analysis_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_schedule_test PASSED in 1.4s //tensorflow/compiler/xla/service:hlo_sharding_test PASSED in 0.8s //tensorflow/compiler/xla/service:hlo_value_semantics_analysis_test PASSED in 1.0s //tensorflow/compiler/xla/service:hlo_verifier_test PASSED in 2.1s //tensorflow/compiler/xla/service:indexed_array_analysis_test PASSED in 1.0s //tensorflow/compiler/xla/service:instruction_fusion_test PASSED in 1.4s //tensorflow/compiler/xla/service:latency_hiding_scheduler_test PASSED in 1.5s //tensorflow/compiler/xla/service:layout_assignment_test PASSED in 6.1s //tensorflow/compiler/xla/service:layout_normalization_test PASSED in 1.9s //tensorflow/compiler/xla/service:logistic_expander_test PASSED in 1.2s //tensorflow/compiler/xla/service:loop_schedule_linearizer_test PASSED in 1.0s //tensorflow/compiler/xla/service:map_inliner_test PASSED in 0.9s //tensorflow/compiler/xla/service:mapped_ptr_container_sorter_test PASSED in 0.1s //tensorflow/compiler/xla/service:memory_space_assignment_best_fit_repacker_test PASSED in 0.2s //tensorflow/compiler/xla/service:memory_space_assignment_test PASSED in 2.6s //tensorflow/compiler/xla/service:memory_space_propagation_test PASSED in 0.7s //tensorflow/compiler/xla/service:name_uniquer_test PASSED in 0.1s //tensorflow/compiler/xla/service:operand_upcaster_test PASSED in 1.5s //tensorflow/compiler/xla/service:optimize_input_output_buffer_alias_test PASSED in 1.4s //tensorflow/compiler/xla/service:pattern_matcher_gmock_test PASSED in 0.2s //tensorflow/compiler/xla/service:pattern_matcher_test PASSED in 1.1s //tensorflow/compiler/xla/service:profile_guided_latency_estimator_test PASSED in 1.1s //tensorflow/compiler/xla/service:real_imag_expander_test PASSED in 1.1s //tensorflow/compiler/xla/service:reduce_decomposer_test PASSED in 1.5s //tensorflow/compiler/xla/service:reduce_scatter_combiner_test PASSED in 1.6s //tensorflow/compiler/xla/service:reduce_scatter_decomposer_test PASSED in 0.8s //tensorflow/compiler/xla/service:reduce_scatter_reassociate_test PASSED in 1.1s //tensorflow/compiler/xla/service:reshape_decomposer_test PASSED in 1.2s //tensorflow/compiler/xla/service:reshape_mover_test PASSED in 1.2s //tensorflow/compiler/xla/service:result_caster_test PASSED in 1.3s //tensorflow/compiler/xla/service:root_instruction_sinker_test PASSED in 0.9s //tensorflow/compiler/xla/service:scatter_expander_test PASSED in 1.1s //tensorflow/compiler/xla/service:scatter_simplifier_test PASSED in 1.1s //tensorflow/compiler/xla/service:select_and_scatter_expander_test PASSED in 1.1s //tensorflow/compiler/xla/service:shape_inference_test PASSED in 0.2s //tensorflow/compiler/xla/service:shaped_buffer_test PASSED in 6.0s //tensorflow/compiler/xla/service:sharding_propagation_test PASSED in 3.2s //tensorflow/compiler/xla/service:sharding_remover_test PASSED in 0.9s //tensorflow/compiler/xla/service:simplify_fp_conversions_test PASSED in 0.9s //tensorflow/compiler/xla/service:slice_sinker_test PASSED in 1.2s //tensorflow/compiler/xla/service:sort_simplifier_test PASSED in 0.8s //tensorflow/compiler/xla/service:space_to_batch_converter_test PASSED in 0.9s //tensorflow/compiler/xla/service:stable_sort_expander_test PASSED in 0.9s //tensorflow/compiler/xla/service:stochastic_convert_decomposer_test PASSED in 1.0s //tensorflow/compiler/xla/service:stream_pool_test PASSED in 0.2s //tensorflow/compiler/xla/service:topk_rewriter_test PASSED in 3.7s //tensorflow/compiler/xla/service:transpose_folding_test PASSED in 1.6s //tensorflow/compiler/xla/service:tuple_points_to_analysis_test PASSED in 1.0s //tensorflow/compiler/xla/service:tuple_simplifier_test PASSED in 0.9s //tensorflow/compiler/xla/service:tuple_util_test PASSED in 1.0s //tensorflow/compiler/xla/service:while_loop_all_reduce_code_motion_test PASSED in 1.7s //tensorflow/compiler/xla/service:while_loop_analysis_test PASSED in 0.9s //tensorflow/compiler/xla/service:while_loop_concat_code_motion_test PASSED in 1.0s //tensorflow/compiler/xla/service:while_loop_constant_sinking_test PASSED in 0.9s //tensorflow/compiler/xla/service:while_loop_expensive_invariant_code_motion_test PASSED in 1.1s //tensorflow/compiler/xla/service:while_loop_invariant_code_motion_test PASSED in 1.1s //tensorflow/compiler/xla/service:while_loop_simplifier_test PASSED in 1.3s //tensorflow/compiler/xla/service:while_loop_trip_count_annotator_test PASSED in 0.8s //tensorflow/compiler/xla/service:while_util_test PASSED in 1.0s //tensorflow/compiler/xla/service:xla_aot_compile_stablehlo_cpu_test PASSED in 7.8s //tensorflow/compiler/xla/service:xla_debug_info_manager_test PASSED in 0.8s //tensorflow/compiler/xla/service:zero_sized_hlo_elimination_test PASSED in 0.8s //tensorflow/compiler/xla/service/cpu:conv_canonicalization_test PASSED in 1.3s //tensorflow/compiler/xla/service/cpu:cpu_eigen_tensor_alignment_test PASSED in 1.1s //tensorflow/compiler/xla/service/cpu:cpu_instruction_fusion_test PASSED in 1.5s //tensorflow/compiler/xla/service/cpu:cpu_layout_assignment_test PASSED in 2.4s //tensorflow/compiler/xla/service/cpu:ir_emission_utils_test PASSED in 1.3s //tensorflow/compiler/xla/service/cpu:parallel_task_assignment_test PASSED in 2.7s //tensorflow/compiler/xla/service/cpu:runtime_fft_test PASSED in 0.1s //tensorflow/compiler/xla/service/cpu:shape_partition_test PASSED in 0.9s //tensorflow/compiler/xla/service/cpu:xfeed_manager_test PASSED in 0.8s //tensorflow/compiler/xla/service/cpu/tests:cpu_bytesizeof_test PASSED in 0.8s //tensorflow/compiler/xla/service/cpu/tests:cpu_dyn_shape_test PASSED in 7.0s //tensorflow/compiler/xla/service/cpu/tests:cpu_eigen_dot_operation_test PASSED in 6.3s //tensorflow/compiler/xla/service/cpu/tests:cpu_external_constants_test PASSED in 21.7s //tensorflow/compiler/xla/service/cpu/tests:cpu_fusion_test PASSED in 6.1s //tensorflow/compiler/xla/service/cpu/tests:cpu_infeed_test PASSED in 7.9s //tensorflow/compiler/xla/service/cpu/tests:cpu_intrinsic_test PASSED in 9.8s //tensorflow/compiler/xla/service/cpu/tests:cpu_key_value_sort_test PASSED in 7.8s //tensorflow/compiler/xla/service/cpu/tests:cpu_literal_caching_test PASSED in 6.8s //tensorflow/compiler/xla/service/cpu/tests:cpu_noalias_test PASSED in 6.6s //tensorflow/compiler/xla/service/cpu/tests:cpu_outfeed_test PASSED in 7.3s //tensorflow/compiler/xla/service/cpu/tests:cpu_profiling_test PASSED in 8.7s //tensorflow/compiler/xla/service/cpu/tests:cpu_spmd_compile_test PASSED in 6.7s //tensorflow/compiler/xla/service/cpu/tests:cpu_topk_test PASSED in 9.3s //tensorflow/compiler/xla/service/cpu/tests:cpu_vectorization_test PASSED in 8.6s //tensorflow/compiler/xla/service/cpu/tests:cpu_while_test PASSED in 7.7s //tensorflow/compiler/xla/service/cpu/tests:tree_reduction_rewriter_test PASSED in 6.8s //tensorflow/compiler/xla/service/gpu:alias_passthrough_params_test PASSED in 1.0s //tensorflow/compiler/xla/service/gpu:all_reduce_blueconnect_test PASSED in 0.8s //tensorflow/compiler/xla/service/gpu:cublas_pad_for_gemms_test PASSED in 1.5s //tensorflow/compiler/xla/service/gpu:cudnn_pad_for_convolutions_test PASSED in 1.6s //tensorflow/compiler/xla/service/gpu:cudnn_simplify_padding_test PASSED in 1.6s //tensorflow/compiler/xla/service/gpu:cudnn_support_utils_test PASSED in 1.4s //tensorflow/compiler/xla/service/gpu:cudnn_vectorize_convolutions_test PASSED in 0.9s //tensorflow/compiler/xla/service/gpu:fusion_merger_test PASSED in 1.5s //tensorflow/compiler/xla/service/gpu:gemm_rewriter_triton_test PASSED in 2.0s //tensorflow/compiler/xla/service/gpu:gpu_conv_padding_legalization_test PASSED in 0.8s //tensorflow/compiler/xla/service/gpu:gpu_conv_rewriter_test PASSED in 0.9s //tensorflow/compiler/xla/service/gpu:gpu_fusible_test PASSED in 1.8s //tensorflow/compiler/xla/service/gpu:gpu_hlo_cost_analysis_test PASSED in 1.4s //tensorflow/compiler/xla/service/gpu:gpu_performance_model_test PASSED in 1.9s //tensorflow/compiler/xla/service/gpu:gpu_sanitize_constant_names_test PASSED in 1.5s //tensorflow/compiler/xla/service/gpu:hlo_algorithm_denylist_test PASSED in 0.2s //tensorflow/compiler/xla/service/gpu:hlo_fusion_stats_test PASSED in 1.1s //tensorflow/compiler/xla/service/gpu:instruction_fusion_test PASSED in 1.8s //tensorflow/compiler/xla/service/gpu:ir_emission_utils_test PASSED in 1.6s //tensorflow/compiler/xla/service/gpu:matmul_utils_test PASSED in 1.8s //tensorflow/compiler/xla/service/gpu:move_copy_to_users_test PASSED in 1.5s //tensorflow/compiler/xla/service/gpu:multi_output_fusion_test PASSED in 2.0s //tensorflow/compiler/xla/service/gpu:non_atomically_upgradeable_rw_lock_test PASSED in 0.2s //tensorflow/compiler/xla/service/gpu:reduction_splitter_test PASSED in 1.7s //tensorflow/compiler/xla/service/gpu:scatter_slice_simplifier_test PASSED in 1.1s //tensorflow/compiler/xla/service/gpu:target_util_test PASSED in 0.5s //tensorflow/compiler/xla/service/gpu:variadic_op_splitter_test PASSED in 1.4s //tensorflow/compiler/xla/service/gpu:while_transformer_test PASSED in 1.6s //tensorflow/compiler/xla/service/gpu/llvm_gpu_backend:utils_test PASSED in 0.9s //tensorflow/compiler/xla/service/gpu/tests:gpu_reduce_scatter_creator_test PASSED in 1.0s //tensorflow/compiler/xla/service/gpu/tests:reduction_degenerate_dim_remover_test PASSED in 1.9s //tensorflow/compiler/xla/service/gpu/tests:reduction_dimension_grouper_test PASSED in 1.3s //tensorflow/compiler/xla/service/gpu/tests:tree_reduction_rewriter_test PASSED in 1.8s //tensorflow/compiler/xla/service/graphcycles:graphcycles_test PASSED in 0.7s //tensorflow/compiler/xla/service/graphcycles:ordered_set_test PASSED in 0.2s //tensorflow/compiler/xla/service/llvm_ir:alias_analysis_test PASSED in 6.4s //tensorflow/compiler/xla/service/llvm_ir:ir_array_test PASSED in 0.9s //tensorflow/compiler/xla/service/spmd:canonicalize_all_gather_for_cse_test PASSED in 1.1s //tensorflow/compiler/xla/service/spmd:collective_permute_motion_test PASSED in 1.0s //tensorflow/compiler/xla/service/spmd:partition_assignment_test PASSED in 1.3s //tensorflow/compiler/xla/service/spmd:schedule_aware_collective_ops_cse_test PASSED in 1.1s //tensorflow/compiler/xla/service/spmd:spmd_partitioner_test PASSED in 2.9s //tensorflow/compiler/xla/service/spmd:stateful_rng_spmd_partitioner_test PASSED in 1.1s //tensorflow/compiler/xla/stream_executor:dnn_test PASSED in 0.1s //tensorflow/compiler/xla/stream_executor:stream_test PASSED in 0.1s //tensorflow/compiler/xla/stream_executor/host:host_stream_test PASSED in 0.3s //tensorflow/compiler/xla/tests:all_reduce_test_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests:axpy_simple_test_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests:bad_rng_shape_validation_test_cpu PASSED in 6.6s //tensorflow/compiler/xla/tests:binop_scaling_test_cpu PASSED in 6.7s //tensorflow/compiler/xla/tests:bitcast_convert_test_cpu PASSED in 8.7s //tensorflow/compiler/xla/tests:broadcast_simple_test_cpu PASSED in 9.4s //tensorflow/compiler/xla/tests:broadcast_test_cpu PASSED in 6.4s //tensorflow/compiler/xla/tests:buffer_donation_test_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests:call_test_cpu PASSED in 8.7s //tensorflow/compiler/xla/tests:check_execution_arity_test_cpu PASSED in 6.6s //tensorflow/compiler/xla/tests:cholesky_test_cpu PASSED in 17.9s //tensorflow/compiler/xla/tests:client_test_cpu PASSED in 9.3s //tensorflow/compiler/xla/tests:collective_ops_test_cpu PASSED in 21.9s //tensorflow/compiler/xla/tests:compilation_cache_test_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests:compute_constant_test_cpu PASSED in 7.5s //tensorflow/compiler/xla/tests:concat_test_cpu PASSED in 8.3s //tensorflow/compiler/xla/tests:constant_reduction_function_test_cpu PASSED in 7.9s //tensorflow/compiler/xla/tests:constants_test_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests:convert_test_cpu PASSED in 8.5s //tensorflow/compiler/xla/tests:copy_test_cpu PASSED in 9.5s //tensorflow/compiler/xla/tests:cpu_gpu_fusion_test_cpu PASSED in 17.5s //tensorflow/compiler/xla/tests:custom_call_test_cpu PASSED in 8.4s //tensorflow/compiler/xla/tests:deallocation_test_cpu PASSED in 8.7s //tensorflow/compiler/xla/tests:deconstruct_tuple_test_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests:deep_graph_test_cpu PASSED in 8.6s //tensorflow/compiler/xla/tests:execution_profile_test_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests:fft_test_cpu PASSED in 8.2s //tensorflow/compiler/xla/tests:float8_test_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests:floor_ceil_test_cpu PASSED in 8.2s //tensorflow/compiler/xla/tests:fmax_fmin_test_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests:gather_operation_test_cpu PASSED in 9.2s //tensorflow/compiler/xla/tests:get_dimension_size_test_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests:half_test_cpu PASSED in 8.8s //tensorflow/compiler/xla/tests:hlo_metadata_test PASSED in 7.4s //tensorflow/compiler/xla/tests:literal_test_util_test PASSED in 5.2s //tensorflow/compiler/xla/tests:local_client_allocation_test_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests:local_client_aot_test PASSED in 0.2s //tensorflow/compiler/xla/tests:log_test_cpu PASSED in 8.9s //tensorflow/compiler/xla/tests:map_test_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests:matrix_ops_simple_test_cpu PASSED in 12.8s //tensorflow/compiler/xla/tests:multidimensional_slice_test_cpu PASSED in 6.8s //tensorflow/compiler/xla/tests:multiple_devices_on_host_test PASSED in 7.3s //tensorflow/compiler/xla/tests:multithreaded_compilation_test_cpu PASSED in 8.9s //tensorflow/compiler/xla/tests:outfeed_in_nested_computation_test_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests:pad_test_cpu PASSED in 8.5s //tensorflow/compiler/xla/tests:pred_test_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests:query_inferred_shape_test_cpu PASSED in 8.0s //tensorflow/compiler/xla/tests:reduce_hlo_test_cpu PASSED in 9.0s //tensorflow/compiler/xla/tests:reduce_precision_test_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests:replay_test_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests:reshape_motion_test_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests:reverse_test_cpu PASSED in 8.8s //tensorflow/compiler/xla/tests:round_trip_packed_literal_test_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests:round_trip_transfer_test_cpu PASSED in 8.0s //tensorflow/compiler/xla/tests:sample_text_test_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests:scatter_test_cpu PASSED in 19.7s //tensorflow/compiler/xla/tests:select_test_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests:test_utils_test_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests:token_hlo_test_cpu PASSED in 7.0s //tensorflow/compiler/xla/tests:transfer_manager_test_cpu PASSED in 13.5s //tensorflow/compiler/xla/tests:transpose_test_cpu PASSED in 8.2s //tensorflow/compiler/xla/tests:tuple_test_cpu PASSED in 8.3s //tensorflow/compiler/xla/tests:unary_op_test_cpu PASSED in 7.2s //tensorflow/compiler/xla/tests:value_inference_test_cpu PASSED in 6.5s //tensorflow/compiler/xla/tests:vector_ops_reduce_test_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests:vector_ops_simple_test_cpu PASSED in 7.9s //tensorflow/compiler/xla/tests:while_test_cpu PASSED in 9.1s //tensorflow/compiler/xla/tests/fuzz:rand_0_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests/fuzz:rand_10_cpu PASSED in 8.8s //tensorflow/compiler/xla/tests/fuzz:rand_11_cpu PASSED in 7.9s //tensorflow/compiler/xla/tests/fuzz:rand_12_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests/fuzz:rand_13_cpu PASSED in 8.7s //tensorflow/compiler/xla/tests/fuzz:rand_14_cpu PASSED in 7.2s //tensorflow/compiler/xla/tests/fuzz:rand_15_cpu PASSED in 6.7s //tensorflow/compiler/xla/tests/fuzz:rand_16_cpu PASSED in 6.7s //tensorflow/compiler/xla/tests/fuzz:rand_17_cpu PASSED in 7.0s //tensorflow/compiler/xla/tests/fuzz:rand_18_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests/fuzz:rand_19_cpu PASSED in 6.6s //tensorflow/compiler/xla/tests/fuzz:rand_20_cpu PASSED in 6.8s //tensorflow/compiler/xla/tests/fuzz:rand_21_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests/fuzz:rand_22_cpu PASSED in 7.2s //tensorflow/compiler/xla/tests/fuzz:rand_23_cpu PASSED in 6.8s //tensorflow/compiler/xla/tests/fuzz:rand_24_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests/fuzz:rand_25_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests/fuzz:rand_26_cpu PASSED in 6.8s //tensorflow/compiler/xla/tests/fuzz:rand_27_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests/fuzz:rand_28_cpu PASSED in 7.5s //tensorflow/compiler/xla/tests/fuzz:rand_29_cpu PASSED in 6.6s //tensorflow/compiler/xla/tests/fuzz:rand_2_cpu PASSED in 7.0s //tensorflow/compiler/xla/tests/fuzz:rand_30_cpu PASSED in 6.5s //tensorflow/compiler/xla/tests/fuzz:rand_31_cpu PASSED in 10.2s //tensorflow/compiler/xla/tests/fuzz:rand_32_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests/fuzz:rand_33_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests/fuzz:rand_34_cpu PASSED in 8.0s //tensorflow/compiler/xla/tests/fuzz:rand_35_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests/fuzz:rand_36_cpu PASSED in 6.8s //tensorflow/compiler/xla/tests/fuzz:rand_37_cpu PASSED in 10.8s //tensorflow/compiler/xla/tests/fuzz:rand_38_cpu PASSED in 8.7s //tensorflow/compiler/xla/tests/fuzz:rand_39_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests/fuzz:rand_3_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests/fuzz:rand_40_cpu PASSED in 7.5s //tensorflow/compiler/xla/tests/fuzz:rand_41_cpu PASSED in 6.2s //tensorflow/compiler/xla/tests/fuzz:rand_42_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests/fuzz:rand_43_cpu PASSED in 8.2s //tensorflow/compiler/xla/tests/fuzz:rand_44_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests/fuzz:rand_45_cpu PASSED in 7.0s //tensorflow/compiler/xla/tests/fuzz:rand_46_cpu PASSED in 6.5s //tensorflow/compiler/xla/tests/fuzz:rand_47_cpu PASSED in 7.6s //tensorflow/compiler/xla/tests/fuzz:rand_48_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests/fuzz:rand_49_cpu PASSED in 8.6s //tensorflow/compiler/xla/tests/fuzz:rand_4_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests/fuzz:rand_50_cpu PASSED in 8.0s //tensorflow/compiler/xla/tests/fuzz:rand_51_cpu PASSED in 6.8s //tensorflow/compiler/xla/tests/fuzz:rand_52_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests/fuzz:rand_53_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests/fuzz:rand_54_cpu PASSED in 6.8s //tensorflow/compiler/xla/tests/fuzz:rand_56_cpu PASSED in 6.6s //tensorflow/compiler/xla/tests/fuzz:rand_57_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests/fuzz:rand_58_cpu PASSED in 8.3s //tensorflow/compiler/xla/tests/fuzz:rand_59_cpu PASSED in 6.4s //tensorflow/compiler/xla/tests/fuzz:rand_5_cpu PASSED in 6.6s //tensorflow/compiler/xla/tests/fuzz:rand_61_cpu PASSED in 8.0s //tensorflow/compiler/xla/tests/fuzz:rand_62_cpu PASSED in 9.0s //tensorflow/compiler/xla/tests/fuzz:rand_63_cpu PASSED in 7.9s //tensorflow/compiler/xla/tests/fuzz:rand_64_cpu PASSED in 7.0s //tensorflow/compiler/xla/tests/fuzz:rand_65_cpu PASSED in 6.6s //tensorflow/compiler/xla/tests/fuzz:rand_66_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests/fuzz:rand_68_cpu PASSED in 6.7s //tensorflow/compiler/xla/tests/fuzz:rand_69_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests/fuzz:rand_6_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests/fuzz:rand_70_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests/fuzz:rand_71_cpu PASSED in 7.2s //tensorflow/compiler/xla/tests/fuzz:rand_73_cpu PASSED in 6.8s //tensorflow/compiler/xla/tests/fuzz:rand_74_cpu PASSED in 7.0s //tensorflow/compiler/xla/tests/fuzz:rand_75_cpu PASSED in 7.0s //tensorflow/compiler/xla/tests/fuzz:rand_76_cpu PASSED in 8.1s //tensorflow/compiler/xla/tests/fuzz:rand_77_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests/fuzz:rand_78_cpu PASSED in 6.7s //tensorflow/compiler/xla/tests/fuzz:rand_79_cpu PASSED in 7.5s //tensorflow/compiler/xla/tests/fuzz:rand_7_cpu PASSED in 6.3s //tensorflow/compiler/xla/tests/fuzz:rand_80_cpu PASSED in 6.5s //tensorflow/compiler/xla/tests/fuzz:rand_81_cpu PASSED in 7.3s //tensorflow/compiler/xla/tests/fuzz:rand_82_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests/fuzz:rand_83_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests/fuzz:rand_84_cpu PASSED in 7.0s //tensorflow/compiler/xla/tests/fuzz:rand_85_cpu PASSED in 7.7s //tensorflow/compiler/xla/tests/fuzz:rand_86_cpu PASSED in 8.1s //tensorflow/compiler/xla/tests/fuzz:rand_87_cpu PASSED in 10.0s //tensorflow/compiler/xla/tests/fuzz:rand_88_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests/fuzz:rand_89_cpu PASSED in 9.9s //tensorflow/compiler/xla/tests/fuzz:rand_8_cpu PASSED in 8.3s //tensorflow/compiler/xla/tests/fuzz:rand_90_cpu PASSED in 6.7s //tensorflow/compiler/xla/tests/fuzz:rand_91_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests/fuzz:rand_92_cpu PASSED in 8.8s //tensorflow/compiler/xla/tests/fuzz:rand_93_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests/fuzz:rand_94_cpu PASSED in 7.1s //tensorflow/compiler/xla/tests/fuzz:rand_95_cpu PASSED in 7.8s //tensorflow/compiler/xla/tests/fuzz:rand_96_cpu PASSED in 8.3s //tensorflow/compiler/xla/tests/fuzz:rand_97_cpu PASSED in 6.9s //tensorflow/compiler/xla/tests/fuzz:rand_98_cpu PASSED in 6.7s //tensorflow/compiler/xla/tests/fuzz:rand_99_cpu PASSED in 7.4s //tensorflow/compiler/xla/tests/fuzz:rand_9_cpu PASSED in 7.6s //tensorflow/compiler/xla/tools:hlo_control_flow_flattening_test PASSED in 1.1s //tensorflow/compiler/xla/tools:hlo_extractor_test PASSED in 0.8s //tensorflow/compiler/xla/tools:hlo_module_loader_test PASSED in 1.1s //tensorflow/compiler/xla/tools:interactive_graphviz_bin_test PASSED in 0.3s //tensorflow/compiler/xla/tools:run_hlo_module_bin_test PASSED in 0.4s //tensorflow/compiler/xla/tools/hlo_bisect:hlo_bisect_state_test PASSED in 0.8s //tensorflow/compiler/xla/translate/hlo_to_mhlo:hlo_utils_test PASSED in 0.8s //tensorflow/compiler/xla/translate/hlo_to_mhlo:mlir_hlo_builder_test PASSED in 0.9s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:bool_compare.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:case_conditional.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:dynamic_param.hlo.test PASSED in 0.7s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:entry_computation_layout.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:frontend_attributes.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fully_connected_reference_model.hlotxt.test PASSED in 0.7s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:fusion.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:if_conditional.hlotxt.test PASSED in 0.7s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import.hlotxt.test PASSED in 0.7s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:import_async.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:layouts_and_names.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:location.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:module_attributes.hlo.test PASSED in 0.6s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:simple.hlo.test PASSED in 0.6s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:spmd_module_sharding.hlo.test PASSED in 0.6s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:types.hlotxt.test PASSED in 0.7s //tensorflow/compiler/xla/translate/hlo_to_mhlo/tests:while.hlotxt.test PASSED in 0.8s //tensorflow/compiler/xla/translate/mhlo_to_hlo:type_to_shape_test PASSED in 1.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:add.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:case.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:dynamic.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export-with-layouts.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export.mlir.test PASSED in 1.4s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_and_check_layouts.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_large_constants.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:export_replicas.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:frontend_attributes.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:fusion.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:if.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:input_output_aliasing.mlir.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:layouts_and_names.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:location_to_op_metadata.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:missing_main.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:module_attributes.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:multiple_return_tuple.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:opaque_elements_attr.mlir.test PASSED in 0.4s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:rng_get_and_update_state.mlir.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:sharding.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:simple.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:unsupported_type.mlir.test PASSED in 0.6s //tensorflow/compiler/xla/translate/mhlo_to_hlo/tests:while.mlir.test PASSED in 0.8s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:hlo_text_to_lhlo_no_opt.hlotxt.test PASSED in 1.7s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:no_opt_ops.hlotxt.test PASSED in 0.5s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:non_identity_layouts.hlotxt.test PASSED in 0.7s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:ops.mlir.test PASSED in 3.0s //tensorflow/compiler/xla/translate/mhlo_to_lhlo_with_xla/tests:passthrough.mlir.test PASSED in 0.8s //tensorflow/core:__tensorflow_core_lib_core_legacy_lib_core_all_tests PASSED in 11.1s //tensorflow/core:__tensorflow_core_lib_gtl_legacy_lib_gtl_tests PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_monitoring_cell_reader_test PASSED in 38.6s //tensorflow/core:__tensorflow_core_lib_monitoring_collection_registry_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_monitoring_counter_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_gauge_test PASSED in 0.3s //tensorflow/core:__tensorflow_core_lib_monitoring_metric_def_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_percentile_sampler_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_sampler_test PASSED in 0.1s //tensorflow/core:__tensorflow_core_lib_monitoring_test_utils_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_lib_strings_legacy_low_level_library_tests PASSED in 0.7s //tensorflow/core:__tensorflow_core_lib_wav_wav_io_test PASSED in 0.2s //tensorflow/core:__tensorflow_core_util_mkl_util_test_srcs PASSED in 0.3s //tensorflow/core:__tensorflow_tsl_lib_core_legacy_lib_core_all_tests PASSED in 0.4s //tensorflow/core:lib_strings_ordered_code_test PASSED in 1.6s //tensorflow/core:lib_strings_proto_serialization_test PASSED in 0.1s //tensorflow/core/api_def:api_test PASSED in 3.6s //tensorflow/core/api_def:update_api_def_test PASSED in 0.2s //tensorflow/core/common_runtime:all_to_all_test_cpu PASSED in 0.9s //tensorflow/core/common_runtime:arg_ret_placement_test PASSED in 0.5s //tensorflow/core/common_runtime:buf_rendezvous_test PASSED in 0.9s //tensorflow/core/common_runtime:collective_executor_mgr_test PASSED in 0.8s //tensorflow/core/common_runtime:collective_param_resolver_local_test PASSED in 5.6s //tensorflow/core/common_runtime:collective_rma_local_test PASSED in 0.8s //tensorflow/core/common_runtime:composite_device_test PASSED in 0.5s //tensorflow/core/common_runtime:cost_measurement_registry_test PASSED in 2.6s //tensorflow/core/common_runtime:cost_util_test PASSED in 0.1s //tensorflow/core/common_runtime:device_mgr_test PASSED in 0.9s //tensorflow/core/common_runtime:device_propagation_test PASSED in 0.7s //tensorflow/core/common_runtime:device_resolver_local_test PASSED in 1.0s //tensorflow/core/common_runtime:device_set_test PASSED in 1.0s //tensorflow/core/common_runtime:direct_session_test_cpu PASSED in 1.4s //tensorflow/core/common_runtime:direct_session_with_debug_test PASSED in 2.7s //tensorflow/core/common_runtime:direct_session_with_tracking_alloc_test PASSED in 1.5s //tensorflow/core/common_runtime:dynamic_device_mgr_test PASSED in 0.9s //tensorflow/core/common_runtime:eval_const_tensor_test PASSED in 0.7s //tensorflow/core/common_runtime:executor_test PASSED in 1.5s //tensorflow/core/common_runtime:function_optimization_registration_test PASSED in 0.9s //tensorflow/core/common_runtime:function_optimization_registry_no_pass_test PASSED in 0.8s //tensorflow/core/common_runtime:function_optimization_registry_pass_failure_test PASSED in 0.7s //tensorflow/core/common_runtime:function_optimization_registry_test PASSED in 1.0s //tensorflow/core/common_runtime:function_threadpool_test PASSED in 1.0s //tensorflow/core/common_runtime:graph_constructor_test PASSED in 2.4s //tensorflow/core/common_runtime:graph_runner_test PASSED in 0.9s //tensorflow/core/common_runtime:hierarchical_tree_broadcaster_test_cpu PASSED in 3.8s //tensorflow/core/common_runtime:inline_function_utils_test PASSED in 4.2s //tensorflow/core/common_runtime:input_colocation_exemption_registry_test PASSED in 0.6s //tensorflow/core/common_runtime:int32_fulltype_test PASSED in 0.7s //tensorflow/core/common_runtime:isolate_placer_inspection_required_ops_pass_test PASSED in 0.9s //tensorflow/core/common_runtime:lower_case_op_test PASSED in 2.4s //tensorflow/core/common_runtime:lower_function_call_test PASSED in 2.4s //tensorflow/core/common_runtime:lower_functional_ops_test PASSED in 2.5s //tensorflow/core/common_runtime:lower_if_op_test PASSED in 2.8s //tensorflow/core/common_runtime:lower_while_op_test PASSED in 2.0s //tensorflow/core/common_runtime:mkl_cpu_allocator_test PASSED in 0.1s //tensorflow/core/common_runtime:mkl_threadpool_device_test PASSED in 0.2s //tensorflow/core/common_runtime:no_op_cost_measurement_test PASSED in 0.2s //tensorflow/core/common_runtime:null_request_cost_accessor_test PASSED in 0.1s //tensorflow/core/common_runtime:optimization_registry_test PASSED in 0.8s //tensorflow/core/common_runtime:optimize_cross_host_control_deps_test PASSED in 11.9s //tensorflow/core/common_runtime:optimize_function_graph_utils_test PASSED in 0.6s //tensorflow/core/common_runtime:partitioning_utils_test PASSED in 0.8s //tensorflow/core/common_runtime:pending_counts_test PASSED in 0.7s //tensorflow/core/common_runtime:permuter_test_cpu PASSED in 3.7s //tensorflow/core/common_runtime:placer_inspection_required_ops_utils_test PASSED in 1.0s //tensorflow/core/common_runtime:placer_test PASSED in 0.8s //tensorflow/core/common_runtime:process_function_library_runtime_test_cpu PASSED in 0.6s //tensorflow/core/common_runtime:process_util_test PASSED in 0.3s //tensorflow/core/common_runtime:quantize_training_test PASSED in 2.2s //tensorflow/core/common_runtime:rendezvous_util_test PASSED in 0.1s //tensorflow/core/common_runtime:replicate_per_replica_nodes_test PASSED in 0.6s //tensorflow/core/common_runtime:request_cost_accessor_registry_test PASSED in 2.3s //tensorflow/core/common_runtime:request_cost_test PASSED in 0.3s //tensorflow/core/common_runtime:ring_gatherer_test_cpu PASSED in 3.1s //tensorflow/core/common_runtime:ring_reducer_test_cpu PASSED in 7.0s //tensorflow/core/common_runtime:scoped_allocator_mgr_test PASSED in 5.3s //tensorflow/core/common_runtime:session_test PASSED in 0.7s //tensorflow/core/common_runtime:shape_refiner_test PASSED in 0.8s //tensorflow/core/common_runtime:single_threaded_executor_test PASSED in 1.4s //tensorflow/core/common_runtime:threadpool_device_test PASSED in 0.8s //tensorflow/core/common_runtime:type_inference_test PASSED in 3.6s //tensorflow/core/common_runtime/eager:attr_builder_test PASSED in 27.2s //tensorflow/core/common_runtime/eager:context_test PASSED in 16.8s //tensorflow/core/common_runtime/eager:custom_device_test PASSED in 14.0s //tensorflow/core/common_runtime/eager:eager_executor_test PASSED in 15.1s //tensorflow/core/common_runtime/eager:eager_op_rewrite_registry_test PASSED in 1.1s //tensorflow/core/common_runtime/eager:eager_operation_test PASSED in 12.6s //tensorflow/core/common_runtime/eager:execute_node_test PASSED in 13.5s //tensorflow/core/common_runtime/eager:execute_test PASSED in 32.7s //tensorflow/core/common_runtime/eager:kernel_and_device_test PASSED in 1.0s //tensorflow/core/common_runtime/eager:mkl_eager_op_rewrite_test PASSED in 12.7s //tensorflow/core/common_runtime/eager:placement_test PASSED in 14.2s //tensorflow/core/common_runtime/eager:placement_utils_test PASSED in 14.3s //tensorflow/core/common_runtime/eager:tensor_handle_data_test PASSED in 12.8s //tensorflow/core/common_runtime/eager:tensor_handle_test PASSED in 14.7s //tensorflow/core/common_runtime/gpu:gpu_device_on_non_gpu_machine_test PASSED in 0.1s //tensorflow/core/common_runtime/next_pluggable_device/c:plugin_c_api_test PASSED in 32.7s //tensorflow/core/config:flags_py_test PASSED in 6.1s //tensorflow/core/config:flags_test PASSED in 0.1s //tensorflow/core/data:compression_utils_test PASSED in 2.0s //tensorflow/core/data:dataset_utils_test PASSED in 0.6s //tensorflow/core/data:hash_utils_test PASSED in 0.9s //tensorflow/core/data:metric_utils_test PASSED in 5.8s //tensorflow/core/data:name_utils_test PASSED in 0.1s //tensorflow/core/data:rewrite_utils_test PASSED in 0.6s //tensorflow/core/data:serialization_utils_test PASSED in 0.6s //tensorflow/core/data:snapshot_utils_test PASSED in 0.7s //tensorflow/core/data:split_utils_test PASSED in 0.7s //tensorflow/core/data:standalone_save_restore_test PASSED in 2.0s //tensorflow/core/data:standalone_test PASSED in 1.9s //tensorflow/core/data:tfdataz_metrics_test PASSED in 1.5s //tensorflow/core/data:unbounded_thread_pool_test PASSED in 0.4s //tensorflow/core/data/service:auto_shard_rewriter_test PASSED in 1.0s //tensorflow/core/data/service:common_test PASSED in 0.2s //tensorflow/core/data/service:credentials_factory_test PASSED in 0.7s //tensorflow/core/data/service:cross_trainer_cache_test PASSED in 1.5s //tensorflow/core/data/service:data_service_test PASSED in 13.5s //tensorflow/core/data/service:data_transfer_test PASSED in 0.5s //tensorflow/core/data/service:dataset_store_test PASSED in 0.7s //tensorflow/core/data/service:dispatcher_client_test PASSED in 4.9s //tensorflow/core/data/service:dispatcher_state_test PASSED in 0.5s //tensorflow/core/data/service:grpc_dispatcher_impl_test PASSED in 2.0s //tensorflow/core/data/service:grpc_util_test PASSED in 0.8s //tensorflow/core/data/service:grpc_worker_impl_test PASSED in 3.6s //tensorflow/core/data/service:journal_test PASSED in 1.0s //tensorflow/core/data/service:logging_utils_test PASSED in 0.1s //tensorflow/core/data/service:task_runner_test PASSED in 3.3s //tensorflow/core/data/service:test_util_test PASSED in 2.2s //tensorflow/core/data/service:url_test PASSED in 0.1s //tensorflow/core/data/service:utils_test PASSED in 0.5s //tensorflow/core/data/service:validate_utils_test PASSED in 0.2s //tensorflow/core/data/service:worker_client_test PASSED in 6.8s //tensorflow/core/data/service:worker_impl_test PASSED in 3.2s //tensorflow/core/data/service/client:data_service_client_test PASSED in 2.6s //tensorflow/core/data/service/client:utils_test PASSED in 3.5s //tensorflow/core/data/service/client:validate_utils_test PASSED in 2.7s //tensorflow/core/data/service/snapshot:distributed_snapshot_test PASSED in 19.5s //tensorflow/core/data/service/snapshot:file_utils_test PASSED in 0.7s //tensorflow/core/data/service/snapshot:path_utils_test PASSED in 0.3s //tensorflow/core/data/service/snapshot:snapshot_manager_test PASSED in 4.1s //tensorflow/core/data/service/snapshot:snapshot_split_provider_test PASSED in 0.6s //tensorflow/core/data/service/snapshot:snapshot_stream_writer_checkpoint_test PASSED in 4.1s //tensorflow/core/data/service/snapshot:snapshot_stream_writer_test PASSED in 3.1s //tensorflow/core/data/service/snapshot:utils_test PASSED in 0.3s //tensorflow/core/debug:debug_graph_utils_test PASSED in 0.5s //tensorflow/core/distributed_runtime:call_options_test PASSED in 0.2s //tensorflow/core/distributed_runtime:cluster_function_library_runtime_test PASSED in 4.5s //tensorflow/core/distributed_runtime:collective_param_resolver_distributed_test PASSED in 0.8s //tensorflow/core/distributed_runtime:collective_rma_distributed_test PASSED in 0.5s //tensorflow/core/distributed_runtime:device_resolver_distributed_test PASSED in 0.5s //tensorflow/core/distributed_runtime:message_wrappers_test PASSED in 0.2s //tensorflow/core/distributed_runtime:partial_run_mgr_test PASSED in 0.4s //tensorflow/core/distributed_runtime:recent_request_ids_test PASSED in 0.2s //tensorflow/core/distributed_runtime:request_id_test PASSED in 0.2s //tensorflow/core/distributed_runtime:rpc_collective_executor_mgr_test PASSED in 0.5s //tensorflow/core/distributed_runtime:server_lib_test PASSED in 0.4s //tensorflow/core/distributed_runtime:session_mgr_test PASSED in 0.7s //tensorflow/core/distributed_runtime:tensor_coding_test PASSED in 0.1s //tensorflow/core/distributed_runtime/coordination:coordination_service_barrier_proxy_test PASSED in 2.6s //tensorflow/core/distributed_runtime/eager:eager_service_impl_test PASSED in 20.6s //tensorflow/core/distributed_runtime/eager:remote_mgr_test PASSED in 13.9s //tensorflow/core/distributed_runtime/integration_test:c_api_coordination_test_cpu PASSED in 43.8s //tensorflow/core/distributed_runtime/integration_test:c_api_multi_client_test_cpu PASSED in 30.1s //tensorflow/core/distributed_runtime/integration_test:c_api_recoverable_jobs_test_cpu PASSED in 43.5s //tensorflow/core/distributed_runtime/integration_test:c_api_session_coordination_test_cpu PASSED in 30.0s //tensorflow/core/distributed_runtime/rpc:grpc_tensor_coding_test PASSED in 2.8s //tensorflow/core/distributed_runtime/rpc:grpc_worker_cache_test PASSED in 0.8s //tensorflow/core/distributed_runtime/rpc/eager:grpc_eager_client_test PASSED in 0.8s //tensorflow/core/example:example_parser_configuration_test PASSED in 0.7s //tensorflow/core/example:feature_util_test PASSED in 0.1s //tensorflow/core/framework:allocator_test PASSED in 3.8s //tensorflow/core/framework:attr_value_util_test PASSED in 0.8s //tensorflow/core/framework:batch_util_test PASSED in 0.8s //tensorflow/core/framework:bfloat16_test PASSED in 1.2s //tensorflow/core/framework:common_shape_fns_test PASSED in 1.0s //tensorflow/core/framework:dataset_test PASSED in 0.7s //tensorflow/core/framework:device_base_test PASSED in 0.7s //tensorflow/core/framework:disable_jit_test PASSED in 0.8s //tensorflow/core/framework:framework_op_gen_lib_test PASSED in 0.2s //tensorflow/core/framework:framework_op_segment_test PASSED in 0.7s //tensorflow/core/framework:framework_resource_var_test PASSED in 0.4s //tensorflow/core/framework:framework_run_handler_test PASSED in 2.1s //tensorflow/core/framework:framework_run_handler_util_test PASSED in 2.3s //tensorflow/core/framework:full_type_inference_util_test PASSED in 0.7s //tensorflow/core/framework:full_type_util_test PASSED in 1.0s //tensorflow/core/framework:function_test PASSED in 1.1s //tensorflow/core/framework:graph_def_util_test PASSED in 0.7s //tensorflow/core/framework:graph_to_functiondef_test PASSED in 0.8s //tensorflow/core/framework:kernel_def_builder_test PASSED in 0.8s //tensorflow/core/framework:kernel_def_util_test PASSED in 0.8s //tensorflow/core/framework:memory_types_test PASSED in 1.0s //tensorflow/core/framework:model_test PASSED in 1.1s //tensorflow/core/framework:node_def_builder_test PASSED in 0.8s //tensorflow/core/framework:node_def_util_test PASSED in 0.7s //tensorflow/core/framework:node_properties_test PASSED in 0.8s //tensorflow/core/framework:op_compatibility_test PASSED in 0.9s //tensorflow/core/framework:op_def_builder_test PASSED in 0.7s //tensorflow/core/framework:op_def_util_test PASSED in 0.8s //tensorflow/core/framework:op_kernel_test PASSED in 0.8s //tensorflow/core/framework:op_registration_test PASSED in 0.8s //tensorflow/core/framework:partial_tensor_shape_test PASSED in 0.7s //tensorflow/core/framework:rendezvous_test PASSED in 3.1s //tensorflow/core/framework:resource_handle_test PASSED in 0.2s //tensorflow/core/framework:resource_mgr_test PASSED in 1.9s //tensorflow/core/framework:resource_op_kernel_test PASSED in 0.8s //tensorflow/core/framework:shape_inference_test PASSED in 0.7s //tensorflow/core/framework:shape_inference_testutil_test PASSED in 0.9s //tensorflow/core/framework:tensor_shape_test PASSED in 7.3s //tensorflow/core/framework:tensor_slice_test PASSED in 1.6s //tensorflow/core/framework:tensor_test PASSED in 28.2s //tensorflow/core/framework:tensor_testutil_test PASSED in 0.7s //tensorflow/core/framework:tensor_util_test PASSED in 0.9s //tensorflow/core/framework:tracking_allocator_test PASSED in 1.0s //tensorflow/core/framework:types_test PASSED in 0.8s //tensorflow/core/framework:variant_op_registry_test PASSED in 19.4s //tensorflow/core/framework:variant_test PASSED in 0.7s //tensorflow/core/framework/registration:registration_test PASSED in 1.0s //tensorflow/core/function/capture:by_ref_capture_test PASSED in 5.6s //tensorflow/core/function/capture:capture_container_test PASSED in 6.5s //tensorflow/core/function/integration_test:side_inputs_manual_api_test PASSED in 12.2s //tensorflow/core/function/integration_test:side_inputs_test PASSED in 13.6s //tensorflow/core/function/polymorphism:function_cache_test PASSED in 5.7s //tensorflow/core/function/polymorphism:function_type_test PASSED in 5.6s //tensorflow/core/function/polymorphism:type_dispatch_test PASSED in 7.2s //tensorflow/core/function/runtime_client:runtime_client_cc_test PASSED in 50.4s //tensorflow/core/function/trace_type:default_types_test PASSED in 5.3s //tensorflow/core/function/trace_type:serialization_test PASSED in 6.0s //tensorflow/core/function/trace_type:trace_type_test PASSED in 9.9s //tensorflow/core/graph:algorithm_test PASSED in 0.7s //tensorflow/core/graph:collective_order_test PASSED in 0.8s //tensorflow/core/graph:control_flow_test PASSED in 1.0s //tensorflow/core/graph:costmodel_test PASSED in 0.9s //tensorflow/core/graph:edgeset_test PASSED in 0.8s //tensorflow/core/graph:graph_def_builder_test PASSED in 0.8s //tensorflow/core/graph:graph_partition_test PASSED in 0.8s //tensorflow/core/graph:graph_test PASSED in 1.0s //tensorflow/core/graph:node_builder_test PASSED in 0.9s //tensorflow/core/graph:optimizer_cse_test PASSED in 1.2s //tensorflow/core/graph:subgraph_test PASSED in 0.6s //tensorflow/core/graph:tensor_id_test PASSED in 0.9s //tensorflow/core/graph:validate_test PASSED in 0.8s //tensorflow/core/graph/regularization:simple_delete_test PASSED in 0.3s //tensorflow/core/graph/regularization:util_test PASSED in 0.1s //tensorflow/core/grappler:graph_topology_view_test PASSED in 0.1s //tensorflow/core/grappler:graph_view_test PASSED in 2.1s //tensorflow/core/grappler:grappler_item_builder_test PASSED in 2.2s //tensorflow/core/grappler:grappler_item_test PASSED in 1.4s //tensorflow/core/grappler:mutable_graph_view_test PASSED in 1.6s //tensorflow/core/grappler:utils_test PASSED in 2.9s //tensorflow/core/grappler/clusters:single_machine_test PASSED in 24.7s //tensorflow/core/grappler/clusters:virtual_cluster_test PASSED in 1.8s //tensorflow/core/grappler/costs:analytical_cost_estimator_test PASSED in 2.1s //tensorflow/core/grappler/costs:cost_estimator_test PASSED in 0.1s //tensorflow/core/grappler/costs:graph_memory_test PASSED in 1.9s //tensorflow/core/grappler/costs:graph_properties_test PASSED in 4.8s //tensorflow/core/grappler/costs:robust_stats_test PASSED in 0.1s //tensorflow/core/grappler/costs:utils_test PASSED in 2.1s //tensorflow/core/grappler/costs:virtual_placer_test PASSED in 0.3s //tensorflow/core/grappler/costs:virtual_scheduler_test PASSED in 2.0s //tensorflow/core/grappler/graph_analyzer:gen_node_test PASSED in 1.7s //tensorflow/core/grappler/graph_analyzer:graph_analyzer_test PASSED in 1.6s //tensorflow/core/grappler/graph_analyzer:hash_tools_test PASSED in 1.9s //tensorflow/core/grappler/graph_analyzer:sig_node_test PASSED in 2.3s //tensorflow/core/grappler/graph_analyzer:subgraph_test PASSED in 1.6s //tensorflow/core/grappler/inputs:utils_test PASSED in 0.2s //tensorflow/core/grappler/optimizers:arithmetic_optimizer_test_cpu PASSED in 4.2s //tensorflow/core/grappler/optimizers:auto_parallel_test_cpu PASSED in 1.8s //tensorflow/core/grappler/optimizers:common_subgraph_elimination_test_cpu PASSED in 2.9s //tensorflow/core/grappler/optimizers:custom_graph_optimizer_registry_test_cpu PASSED in 4.4s //tensorflow/core/grappler/optimizers:debug_stripper_test_cpu PASSED in 2.7s //tensorflow/core/grappler/optimizers:dependency_optimizer_test_cpu PASSED in 2.4s //tensorflow/core/grappler/optimizers:evaluation_utils_test PASSED in 0.5s //tensorflow/core/grappler/optimizers:function_api_info_test PASSED in 0.1s //tensorflow/core/grappler/optimizers:function_optimizer_test_cpu PASSED in 4.7s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_test_cpu PASSED in 2.0s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_factory_test PASSED in 0.2s //tensorflow/core/grappler/optimizers:generic_layout_optimizer_transposer_test_cpu PASSED in 2.0s //tensorflow/core/grappler/optimizers:graph_optimizer_stage_test_cpu PASSED in 1.9s //tensorflow/core/grappler/optimizers:implementation_selector_test PASSED in 2.5s //tensorflow/core/grappler/optimizers:loop_optimizer_test_cpu PASSED in 2.8s //tensorflow/core/grappler/optimizers:memory_optimizer_test_cpu PASSED in 2.5s //tensorflow/core/grappler/optimizers:meta_optimizer_test_cpu PASSED in 8.1s //tensorflow/core/grappler/optimizers:mkl_remapper_test PASSED in 1.9s //tensorflow/core/grappler/optimizers:model_pruner_test_cpu PASSED in 1.8s //tensorflow/core/grappler/optimizers:pin_to_host_optimizer_test_cpu PASSED in 3.5s //tensorflow/core/grappler/optimizers:scoped_allocator_optimizer_test PASSED in 2.5s //tensorflow/core/grappler/optimizers:shape_optimizer_test_cpu PASSED in 2.0s //tensorflow/core/grappler/optimizers:static_schedule_test_cpu PASSED in 1.6s //tensorflow/core/grappler/optimizers:tfg_optimizer_hook_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:auto_shard_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:autotune_buffer_sizes_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:batch_parallelization_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:disable_intra_op_parallelism_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:disable_prefetch_legacy_autotune_test PASSED in 0.4s //tensorflow/core/grappler/optimizers/data:enable_gradient_descent_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:filter_fusion_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:filter_parallelization_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:function_utils_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:fusion_utils_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:graph_utils_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:inject_prefetch_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:make_deterministic_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:make_sloppy_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:map_and_batch_fusion_test PASSED in 0.4s //tensorflow/core/grappler/optimizers/data:map_and_filter_fusion_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:map_fusion_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:map_parallelization_test PASSED in 1.4s //tensorflow/core/grappler/optimizers/data:noop_elimination_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/data:parallel_batch_test PASSED in 0.7s //tensorflow/core/grappler/optimizers/data:replicate_on_split_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:shuffle_and_repeat_fusion_test PASSED in 0.6s //tensorflow/core/grappler/optimizers/data:slack_test PASSED in 0.8s //tensorflow/core/grappler/optimizers/data:split_utils_test PASSED in 1.5s //tensorflow/core/grappler/optimizers/data:use_private_thread_pool_test PASSED in 0.5s //tensorflow/core/grappler/optimizers/inference:batch_op_rewriter_test PASSED in 0.2s //tensorflow/core/grappler/utils:canonicalizer_test PASSED in 1.5s //tensorflow/core/grappler/utils:colocation_test PASSED in 0.6s //tensorflow/core/grappler/utils:frame_test PASSED in 0.2s //tensorflow/core/grappler/utils:functions_test PASSED in 1.8s //tensorflow/core/grappler/utils:graph_view_internal_test PASSED in 0.6s //tensorflow/core/grappler/utils:graph_view_test PASSED in 1.7s //tensorflow/core/grappler/utils:grappler_test_test PASSED in 7.4s //tensorflow/core/grappler/utils:pattern_utils_test PASSED in 0.6s //tensorflow/core/grappler/utils:scc_test PASSED in 1.6s //tensorflow/core/grappler/utils:symbolic_shapes_test PASSED in 0.2s //tensorflow/core/grappler/utils:topological_sort_test PASSED in 0.6s //tensorflow/core/grappler/utils:tpu_test PASSED in 0.2s //tensorflow/core/grappler/utils:transitive_fanin_test PASSED in 0.5s //tensorflow/core/grappler/utils:traversal_test PASSED in 0.5s //tensorflow/core/grappler/verifiers:structure_verifier_test PASSED in 1.6s //tensorflow/core/ir:interfaces_test PASSED in 0.4s //tensorflow/core/ir:ops_test PASSED in 0.2s //tensorflow/core/ir:shape_inference_utils_test PASSED in 0.3s //tensorflow/core/ir:tf_op_registry_test PASSED in 0.3s //tensorflow/core/ir:tf_op_wrapper_test PASSED in 0.2s //tensorflow/core/ir:utility_test PASSED in 0.3s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:arg_as_control_ret.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:backedge_segment.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:empty.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:error_during_backedge.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_case_with_attr_inference.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_if_with_attr_inference.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_iterator_get_next_attr_inference.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_underscore_output_shapes.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:import_while_with_attr_inference.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infeed_dequeue.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_arg_handle_type.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:infer_with_output_shapes.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_arg_name.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_backedge_input_size.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_duplicated_node_name.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_index.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_edge_name.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_attr_key.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_key.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_func_attr_name.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_empty_op_type.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_func_with_empty_name.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_function_import.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_control_result.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_input.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_name.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_func_with_empty_result.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_attr_name.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_generic_function_named_edge_index.pbtxt.test PASSED in 0.8s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_handle_data.pbtxt.test PASSED in 0.9s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_input.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_control_result_value.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_data_result_value.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_input.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_missing_two_inputs.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_named_edge_index.pbtxt.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_op_name.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:invalid_type_list.pbtxt.test PASSED in 1.0s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:legacy_call.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_shape.pbtxt.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:negative_zero_constant.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:three_nodes_with_attrs.pbtxt.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/graphdef_to_mlir:version.pbtxt.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:empty.mlir.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:fulltype.mlir.test PASSED in 0.7s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:func_with_no_args_or_results.mlir.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:negative_zero_constant.mlir.test PASSED in 0.5s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:nested_legacy_call.mlir.test PASSED in 0.6s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:three_nodes_with_attrs.mlir.test PASSED in 1.4s //tensorflow/core/ir/importexport/tests/mlir_to_graphdef:version.mlir.test PASSED in 1.1s //tensorflow/core/ir/importexport/tests/saved_model:saved_model_roundtrip_test PASSED in 0.3s //tensorflow/core/ir/tests:attributes.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:canonicalize.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:compatible_types.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:concrete-ops.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:generic_concrete_ops.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:invalid-concrete-ops.mlir.test PASSED in 0.5s //tensorflow/core/ir/tests:invalid-preserved-attrs.mlir.test PASSED in 0.8s //tensorflow/core/ir/tests:invalid.mlir.test PASSED in 0.9s //tensorflow/core/ir/tests:invalid_types.mlir.test PASSED in 0.4s //tensorflow/core/ir/tests:ops.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:region-invalid-ops.mlir.test PASSED in 1.0s //tensorflow/core/ir/tests:region-ops-graph.mlir.test PASSED in 0.7s //tensorflow/core/ir/tests:region-ops.mlir.test PASSED in 0.6s //tensorflow/core/ir/tests:types.mlir.test PASSED in 0.9s //tensorflow/core/ir/types:dialect_test PASSED in 0.7s //tensorflow/core/kernels:as_string_op_test PASSED in 1.1s //tensorflow/core/kernels:basic_ops_benchmark_test PASSED in 0.8s //tensorflow/core/kernels:batch_kernels_env_test PASSED in 0.7s //tensorflow/core/kernels:batch_kernels_test PASSED in 1.2s //tensorflow/core/kernels:bias_op_test PASSED in 0.5s //tensorflow/core/kernels:bincount_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:broadcast_to_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:cast_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:checkpoint_callback_manager_test PASSED in 0.9s //tensorflow/core/kernels:clustering_ops_test PASSED in 0.6s //tensorflow/core/kernels:composite_tensor_variant_test PASSED in 0.6s //tensorflow/core/kernels:concat_op_test PASSED in 0.6s //tensorflow/core/kernels:constant_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:control_flow_ops_test PASSED in 7.7s //tensorflow/core/kernels:conv_grad_filter_ops_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels:conv_grad_input_ops_benchmark_test_cpu PASSED in 0.8s //tensorflow/core/kernels:conv_ops_benchmark_test_cpu PASSED in 0.7s //tensorflow/core/kernels:conv_ops_test_cpu PASSED in 7.8s //tensorflow/core/kernels:count_ops_test PASSED in 0.7s //tensorflow/core/kernels:cross_op_test PASSED in 0.6s //tensorflow/core/kernels:cwise_ops_test_cpu PASSED in 0.6s //tensorflow/core/kernels:debug_ops_test PASSED in 1.2s //tensorflow/core/kernels:decode_wav_op_test PASSED in 2.2s //tensorflow/core/kernels:deep_conv2d_test PASSED in 0.3s //tensorflow/core/kernels:dequantize_op_test PASSED in 0.6s //tensorflow/core/kernels:diag_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:dynamic_partition_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:dynamic_stitch_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:eigen_activations_test PASSED in 0.1s //tensorflow/core/kernels:eigen_attention_test PASSED in 0.1s //tensorflow/core/kernels:eigen_backward_cuboid_convolutions_test PASSED in 0.7s //tensorflow/core/kernels:eigen_backward_spatial_convolutions_test PASSED in 0.2s //tensorflow/core/kernels:eigen_benchmark_cpu_test PASSED in 0.1s //tensorflow/core/kernels:eigen_mkldnn_contraction_kernel_test PASSED in 0.1s //tensorflow/core/kernels:eigen_pooling_test PASSED in 0.4s //tensorflow/core/kernels:encode_wav_op_test PASSED in 2.8s //tensorflow/core/kernels:fingerprint_op_test PASSED in 0.8s //tensorflow/core/kernels:fused_batch_norm_ex_op_test_cpu PASSED in 1.1s //tensorflow/core/kernels:fused_batch_norm_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:gather_nd_op_test_cpu PASSED in 0.9s //tensorflow/core/kernels:gather_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:guarantee_const_op_test PASSED in 0.5s //tensorflow/core/kernels:identity_n_op_test PASSED in 0.6s //tensorflow/core/kernels:identity_op_test PASSED in 0.5s //tensorflow/core/kernels:immutable_constant_op_test PASSED in 0.9s //tensorflow/core/kernels:in_topk_op_test PASSED in 0.6s //tensorflow/core/kernels:isotonic_regression_op_test PASSED in 0.6s //tensorflow/core/kernels:logging_ops_test PASSED in 1.7s //tensorflow/core/kernels:lookup_ops_test PASSED in 0.6s //tensorflow/core/kernels:loss_test PASSED in 0.1s //tensorflow/core/kernels:lrn_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:matmul_op_test_cpu PASSED in 4.0s //tensorflow/core/kernels:merge_v2_checkpoints_op_test PASSED in 1.7s //tensorflow/core/kernels:mfcc_dct_test PASSED in 0.1s //tensorflow/core/kernels:mfcc_mel_filterbank_test PASSED in 0.1s //tensorflow/core/kernels:mfcc_op_test_cpu PASSED in 2.0s //tensorflow/core/kernels:mfcc_test PASSED in 0.2s //tensorflow/core/kernels:multinomial_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:nn_ops_test_cpu PASSED in 0.4s //tensorflow/core/kernels:one_hot_op_test PASSED in 0.5s //tensorflow/core/kernels:ops_testutil_test PASSED in 0.5s //tensorflow/core/kernels:ops_util_test PASSED in 0.2s //tensorflow/core/kernels:parameterized_truncated_normal_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:parse_tensor_test PASSED in 0.8s //tensorflow/core/kernels:quantization_utils_test PASSED in 0.6s //tensorflow/core/kernels:quantize_and_dequantize_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:quantize_down_and_shrink_range_op_test PASSED in 0.7s //tensorflow/core/kernels:quantize_op_test PASSED in 1.2s //tensorflow/core/kernels:quantized_activation_ops_test PASSED in 0.7s //tensorflow/core/kernels:quantized_add_op_test PASSED in 1.4s //tensorflow/core/kernels:quantized_batch_norm_op_test PASSED in 0.7s //tensorflow/core/kernels:quantized_bias_add_op_test PASSED in 0.8s //tensorflow/core/kernels:quantized_concat_op_test PASSED in 0.7s //tensorflow/core/kernels:quantized_conv_ops_test PASSED in 0.7s //tensorflow/core/kernels:quantized_instance_norm_test PASSED in 1.0s //tensorflow/core/kernels:quantized_matmul_op_test PASSED in 1.1s //tensorflow/core/kernels:quantized_mul_op_test PASSED in 1.0s //tensorflow/core/kernels:quantized_pooling_ops_test PASSED in 0.6s //tensorflow/core/kernels:quantized_reshape_op_test PASSED in 0.5s //tensorflow/core/kernels:quantized_resize_bilinear_op_test PASSED in 2.1s //tensorflow/core/kernels:ragged_fill_empty_rows_op_test PASSED in 0.7s //tensorflow/core/kernels:ragged_gather_op_test PASSED in 0.6s //tensorflow/core/kernels:ragged_range_op_test PASSED in 0.5s //tensorflow/core/kernels:ragged_tensor_from_variant_op_test PASSED in 0.6s //tensorflow/core/kernels:ragged_tensor_to_sparse_kernel_test PASSED in 0.5s //tensorflow/core/kernels:ragged_tensor_to_tensor_op_test PASSED in 0.9s //tensorflow/core/kernels:ragged_tensor_to_variant_op_test PASSED in 0.7s //tensorflow/core/kernels:random_binomial_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:random_index_shuffle_test PASSED in 0.3s //tensorflow/core/kernels:random_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:random_poisson_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:range_sampler_test PASSED in 0.5s //tensorflow/core/kernels:reduction_ops_test_cpu PASSED in 0.5s //tensorflow/core/kernels:regex_replace_op_test PASSED in 0.6s //tensorflow/core/kernels:requantization_range_op_test PASSED in 0.6s //tensorflow/core/kernels:requantize_op_test PASSED in 0.9s //tensorflow/core/kernels:resource_ops_test PASSED in 0.7s //tensorflow/core/kernels:restore_op_test PASSED in 0.5s //tensorflow/core/kernels:restore_v2_op_test PASSED in 0.5s //tensorflow/core/kernels:reverse_op_test PASSED in 0.8s //tensorflow/core/kernels:roll_op_test PASSED in 1.0s //tensorflow/core/kernels:save_op_test PASSED in 1.3s //tensorflow/core/kernels:save_v2_op_test PASSED in 0.9s //tensorflow/core/kernels:scan_ops_test_cpu PASSED in 0.6s //tensorflow/core/kernels:scatter_nd_op_test_cpu PASSED in 0.7s //tensorflow/core/kernels:scatter_op_test PASSED in 0.6s //tensorflow/core/kernels:scoped_allocator_ops_test_cpu PASSED in 6.9s //tensorflow/core/kernels:sdca_ops_test PASSED in 1.3s //tensorflow/core/kernels:segment_reduction_ops_test PASSED in 0.6s //tensorflow/core/kernels:sendrecv_ops_test PASSED in 0.5s //tensorflow/core/kernels:sequence_ops_test PASSED in 0.5s //tensorflow/core/kernels:shape_ops_test PASSED in 0.7s //tensorflow/core/kernels:slice_op_test PASSED in 0.5s //tensorflow/core/kernels:spacetobatch_benchmark_test_cpu PASSED in 0.6s //tensorflow/core/kernels:sparse_add_op_test PASSED in 0.6s //tensorflow/core/kernels:sparse_dense_binary_op_shared_test PASSED in 1.1s //tensorflow/core/kernels:sparse_fill_empty_rows_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:sparse_matmul_op_test_cpu PASSED in 0.5s //tensorflow/core/kernels:sparse_reduce_sum_op_test PASSED in 0.5s //tensorflow/core/kernels:sparse_tensor_dense_matmul_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:sparse_to_dense_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:sparse_utils_test PASSED in 0.6s //tensorflow/core/kernels:sparse_xent_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels:spectrogram_op_test_cpu PASSED in 2.2s //tensorflow/core/kernels:spectrogram_test PASSED in 0.1s //tensorflow/core/kernels:split_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:split_v_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels:strided_slice_op_test PASSED in 0.5s //tensorflow/core/kernels:string_format_op_test PASSED in 1.2s //tensorflow/core/kernels:string_ngrams_op_test PASSED in 0.8s //tensorflow/core/kernels:string_split_op_test PASSED in 0.5s //tensorflow/core/kernels:substr_op_test PASSED in 0.5s //tensorflow/core/kernels:summary_audio_op_test PASSED in 0.6s //tensorflow/core/kernels:summary_image_op_test PASSED in 0.8s //tensorflow/core/kernels:summary_op_test PASSED in 1.1s //tensorflow/core/kernels:summary_tensor_op_test PASSED in 0.8s //tensorflow/core/kernels:tensor_cord_test PASSED in 0.4s //tensorflow/core/kernels:tensor_flag_utils_test PASSED in 0.1s //tensorflow/core/kernels:tensor_map_test PASSED in 0.1s //tensorflow/core/kernels:training_ops_test PASSED in 0.5s //tensorflow/core/kernels:transpose_util_test PASSED in 0.3s //tensorflow/core/kernels:unary_ops_composition_test_cpu PASSED in 2.3s //tensorflow/core/kernels:unique_op_test PASSED in 1.3s //tensorflow/core/kernels:variable_ops_test PASSED in 1.8s //tensorflow/core/kernels:while_op_test PASSED in 1.0s //tensorflow/core/kernels:xent_op_test_cpu PASSED in 0.8s //tensorflow/core/kernels/batching_util:basic_batch_scheduler_test PASSED in 0.1s //tensorflow/core/kernels/batching_util:batch_input_task_test PASSED in 0.4s //tensorflow/core/kernels/batching_util:batch_resource_base_test PASSED in 0.2s //tensorflow/core/kernels/batching_util:batch_scheduler_test PASSED in 0.4s //tensorflow/core/kernels/batching_util:bounded_executor_test PASSED in 20.1s //tensorflow/core/kernels/batching_util:input_split_metadata_test PASSED in 0.2s //tensorflow/core/kernels/batching_util:periodic_function_test PASSED in 4.0s //tensorflow/core/kernels/batching_util:serial_device_batch_scheduler_test PASSED in 2.4s //tensorflow/core/kernels/batching_util:shared_batch_scheduler_test PASSED in 3.3s //tensorflow/core/kernels/batching_util:threadsafe_status_test PASSED in 0.2s //tensorflow/core/kernels/data:batch_dataset_op_test PASSED in 1.7s //tensorflow/core/kernels/data:cache_dataset_ops_test PASSED in 1.0s //tensorflow/core/kernels/data:concatenate_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:filter_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data:finalize_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:fixed_length_record_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:flat_map_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data:get_options_op_test PASSED in 0.6s //tensorflow/core/kernels/data:interleave_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:iterator_ops_test PASSED in 1.2s //tensorflow/core/kernels/data:map_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:map_defun_op_test PASSED in 0.6s //tensorflow/core/kernels/data:optimize_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:options_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:padded_batch_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:parallel_batch_dataset_op_test PASSED in 1.5s //tensorflow/core/kernels/data:parallel_filter_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:parallel_interleave_dataset_op_test PASSED in 1.4s //tensorflow/core/kernels/data:parallel_map_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:prefetch_autotuner_test PASSED in 0.4s //tensorflow/core/kernels/data:prefetch_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:range_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:reduce_dataset_op_test PASSED in 1.2s //tensorflow/core/kernels/data:repeat_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:rewrite_dataset_op_test PASSED in 1.4s //tensorflow/core/kernels/data:shard_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:shuffle_dataset_op_test PASSED in 1.6s //tensorflow/core/kernels/data:skip_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data:sparse_tensor_slice_dataset_op_test PASSED in 1.1s //tensorflow/core/kernels/data:take_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data:tensor_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:tensor_slice_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data:text_line_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:tf_record_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:window_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data:zip_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data/experimental:assert_next_dataset_op_test PASSED in 0.6s //tensorflow/core/kernels/data/experimental:assert_prev_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/data/experimental:auto_shard_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data/experimental:directed_interleave_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data/experimental:list_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data/experimental:map_and_batch_dataset_op_test PASSED in 1.3s //tensorflow/core/kernels/data/experimental:parallel_interleave_dataset_op_test PASSED in 1.5s //tensorflow/core/kernels/data/experimental:random_dataset_op_test PASSED in 0.7s //tensorflow/core/kernels/data/experimental:sampling_dataset_op_test PASSED in 0.9s //tensorflow/core/kernels/data/experimental:save_dataset_op_test PASSED in 1.0s //tensorflow/core/kernels/data/experimental:unique_dataset_op_test PASSED in 0.8s //tensorflow/core/kernels/image:adjust_contrast_op_benchmark_test_cpu PASSED in 0.6s //tensorflow/core/kernels/image:adjust_contrast_op_test PASSED in 0.7s //tensorflow/core/kernels/image:colorspace_op_test PASSED in 0.6s //tensorflow/core/kernels/image:crop_and_resize_op_benchmark_test_cpu PASSED in 0.9s //tensorflow/core/kernels/image:crop_and_resize_op_test PASSED in 0.6s //tensorflow/core/kernels/image:encode_jpeg_op_test PASSED in 0.6s //tensorflow/core/kernels/image:mirror_pad_op_benchmark_test_cpu PASSED in 0.6s //tensorflow/core/kernels/image:mirror_pad_op_test PASSED in 0.7s //tensorflow/core/kernels/image:non_max_suppression_op_benchmark_test PASSED in 0.8s //tensorflow/core/kernels/image:non_max_suppression_op_test PASSED in 0.8s //tensorflow/core/kernels/image:resize_area_op_test PASSED in 0.9s //tensorflow/core/kernels/image:resize_benchmark_test_cpu PASSED in 0.5s //tensorflow/core/kernels/image:resize_bicubic_op_test PASSED in 3.7s //tensorflow/core/kernels/image:resize_ops_test_cpu PASSED in 3.1s //tensorflow/core/kernels/image:sampling_kernels_test PASSED in 0.5s //tensorflow/core/kernels/image:scale_and_translate_op_test PASSED in 1.6s //tensorflow/core/kernels/linalg:banded_triangular_solve_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels/linalg:matrix_triangular_solve_op_test_cpu PASSED in 0.6s //tensorflow/core/kernels/mkl:mkl_conv_ops_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_dequantize_op_test PASSED in 0.4s //tensorflow/core/kernels/mkl:mkl_fused_batch_norm_op_test PASSED in 0.4s //tensorflow/core/kernels/mkl:mkl_fused_ops_test PASSED in 1.7s //tensorflow/core/kernels/mkl:mkl_matmul_op_benchmark PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_qmatmul_op_test PASSED in 0.4s //tensorflow/core/kernels/mkl:mkl_quantize_op_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_quantized_concat_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_perchannel_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_quantized_conv_ops_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_quantized_pooling_ops_test PASSED in 0.3s //tensorflow/core/kernels/mkl:mkl_relu_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:mkl_requantize_ops_test PASSED in 0.2s //tensorflow/core/kernels/mkl:mkl_swish_op_test PASSED in 0.1s //tensorflow/core/kernels/mkl:onednn_nn_ops_benchmark PASSED in 0.3s //tensorflow/core/kernels/sparse:kernels_test PASSED in 0.7s //tensorflow/core/kernels/uniform_quant_ops:math_utils_test PASSED in 0.2s //tensorflow/core/kernels/uniform_quant_ops:tensor_utils_test PASSED in 0.1s //tensorflow/core/kernels/uniform_quant_ops:uniform_dequantize_op_test PASSED in 0.9s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantize_op_test PASSED in 0.6s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_add_op_test PASSED in 0.9s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_clip_by_value_op_test PASSED in 0.5s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_convolution_ops_test PASSED in 0.7s //tensorflow/core/kernels/uniform_quant_ops:uniform_quantized_dot_ops_test PASSED in 0.6s //tensorflow/core/kernels/uniform_quant_ops:uniform_requantize_op_test PASSED in 1.1s //tensorflow/core/lib/db:sqlite_test PASSED in 0.2s //tensorflow/core/lib/gif:lib_gif_io_test PASSED in 1.2s //tensorflow/core/lib/jpeg:lib_jpeg_jpeg_mem_unittest PASSED in 0.5s //tensorflow/core/ops:cudnn_rnn_ops_test_cc PASSED in 0.5s //tensorflow/core/ops:ops_array_grad_test PASSED in 1.4s //tensorflow/core/ops:ops_math_grad_test PASSED in 4.7s //tensorflow/core/ops:ops_tests PASSED in 0.6s //tensorflow/core/ops/compat:backwards_compatibility_test PASSED in 0.6s //tensorflow/core/platform:__tensorflow_tsl_platform_profile_utils_cpu_utils_test PASSED in 0.2s //tensorflow/core/platform:enable_tf2_utils_test PASSED in 0.1s //tensorflow/core/platform:env_test PASSED in 3.0s //tensorflow/core/platform:fake_python_env_test PASSED in 0.1s //tensorflow/core/platform:file_system_test PASSED in 0.5s //tensorflow/core/platform:platform_strings_test PASSED in 0.1s //tensorflow/core/platform:ram_file_system_test PASSED in 21.1s //tensorflow/core/platform:resource_loader_test PASSED in 0.3s //tensorflow/core/platform:vmodule_benchmark_test PASSED in 0.2s //tensorflow/core/platform:vmodule_test PASSED in 0.4s //tensorflow/core/profiler/backends/cpu:host_tracer_test PASSED in 0.1s //tensorflow/core/profiler/convert:hlo_proto_to_graph_view_test PASSED in 0.2s //tensorflow/core/profiler/convert:hlo_proto_to_memory_visualization_utils_test PASSED in 0.2s //tensorflow/core/profiler/convert:op_stats_to_pod_stats_test PASSED in 0.2s //tensorflow/core/profiler/convert:op_stats_to_pod_viewer_test PASSED in 0.1s //tensorflow/core/profiler/convert:op_stats_to_tf_stats_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_kernel_stats_db_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_memory_profile_test PASSED in 0.3s //tensorflow/core/profiler/convert:xplane_to_op_metrics_db_test PASSED in 0.1s //tensorflow/core/profiler/convert:xplane_to_op_stats_test PASSED in 0.4s //tensorflow/core/profiler/convert:xplane_to_step_events_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_tf_functions_test PASSED in 0.2s //tensorflow/core/profiler/convert:xplane_to_tool_names_test PASSED in 0.1s //tensorflow/core/profiler/convert/trace_viewer:trace_viewer_visibility_test PASSED in 0.1s //tensorflow/core/profiler/internal:tfprof_show_test PASSED in 0.7s //tensorflow/core/profiler/internal:tfprof_stats_test PASSED in 0.7s //tensorflow/core/profiler/internal:tfprof_tensor_test PASSED in 0.6s //tensorflow/core/profiler/internal:tfprof_timeline_test PASSED in 0.6s //tensorflow/core/profiler/internal/advisor:tfprof_advisor_test PASSED in 0.6s //tensorflow/core/profiler/lib:profiler_disabled_test PASSED in 0.2s //tensorflow/core/profiler/utils:derived_timeline_test PASSED in 0.2s //tensorflow/core/profiler/utils:kernel_stats_utils_test PASSED in 0.3s //tensorflow/core/profiler/utils:op_metrics_db_utils_test PASSED in 0.1s //tensorflow/core/profiler/utils:step_intersection_test PASSED in 0.1s //tensorflow/core/summary:schema_test PASSED in 0.2s //tensorflow/core/summary:summary_db_writer_test PASSED in 0.8s //tensorflow/core/summary:summary_file_writer_test PASSED in 0.1s //tensorflow/core/tfrt/common:pjrt_state_test PASSED in 5.4s //tensorflow/core/tfrt/common:pjrt_util_test PASSED in 5.4s //tensorflow/core/tfrt/fallback:cost_recorder_test PASSED in 0.1s //tensorflow/core/tfrt/fallback:fallback_state_test PASSED in 0.4s //tensorflow/core/transforms:eval_utils_test PASSED in 1.9s //tensorflow/core/transforms:graph_transform_wrapper_test PASSED in 0.4s //tensorflow/core/util:bcast_test PASSED in 1.2s //tensorflow/core/util:command_line_flags_test PASSED in 0.9s //tensorflow/core/util:debug_data_dumper_test PASSED in 0.9s //tensorflow/core/util:debug_events_writer_test PASSED in 0.4s //tensorflow/core/util:dump_graph_test PASSED in 0.7s //tensorflow/core/util:equal_graph_def_test PASSED in 0.7s //tensorflow/core/util:events_writer_test PASSED in 2.8s //tensorflow/core/util:example_proto_fast_parsing_test PASSED in 1.1s //tensorflow/core/util:example_proto_helper_test PASSED in 0.8s //tensorflow/core/util:exec_on_stall_test PASSED in 2.2s //tensorflow/core/util:fake_clock_env_test PASSED in 2.2s //tensorflow/core/util:incremental_barrier_test PASSED in 0.1s //tensorflow/core/util:matmul_bcast_test PASSED in 0.7s //tensorflow/core/util:memmapped_file_system_test PASSED in 0.8s //tensorflow/core/util:overflow_test PASSED in 0.2s //tensorflow/core/util:presized_cuckoo_map_test PASSED in 1.7s //tensorflow/core/util:ragged_to_dense_util_test PASSED in 0.5s //tensorflow/core/util:reffed_status_callback_test PASSED in 1.0s //tensorflow/core/util:reporter_test PASSED in 1.0s //tensorflow/core/util:saved_tensor_slice_util_test PASSED in 0.9s //tensorflow/core/util:semver_test PASSED in 0.9s //tensorflow/core/util:stat_summarizer_test PASSED in 0.9s //tensorflow/core/util:strided_slice_op_test PASSED in 0.7s //tensorflow/core/util:tensor_format_test PASSED in 0.6s //tensorflow/core/util:tensor_slice_reader_test PASSED in 1.3s //tensorflow/core/util:tensor_slice_set_test PASSED in 0.8s //tensorflow/core/util:tensor_slice_util_test PASSED in 0.8s //tensorflow/core/util:tensor_slice_writer_test PASSED in 1.4s //tensorflow/core/util:work_sharder_test PASSED in 0.8s //tensorflow/core/util/ctc:ctc_beam_search_test PASSED in 0.2s //tensorflow/core/util/proto:descriptor_pool_registry_test PASSED in 0.6s //tensorflow/core/util/proto:proto_utils_test PASSED in 0.6s //tensorflow/core/util/quantization:uniform_quant_ops_params_test PASSED in 0.6s //tensorflow/core/util/sparse:sparse_tensor_test PASSED in 0.2s //tensorflow/core/util/tensor_bundle:tensor_bundle_test PASSED in 28.6s //tensorflow/dtensor/mlir:dtensor_location_test PASSED in 0.3s //tensorflow/dtensor/mlir:group_assignment_test PASSED in 0.1s //tensorflow/dtensor/mlir/tests:annotate_global_shape.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:cluster_function_conversion.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:constant_folding.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:designate_resource_handle_mesh.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:device_mesh_cluster_coarsening.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:dtensor_all_gather.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_all_scatter.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_combine_optimization.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_lowering.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_scatter_optimization.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:dtensor_allreduce_sum_optimization.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_alltoall_lowering.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:dtensor_layout_must_execute.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_layout_to_xla_sharding_op.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:dtensor_mixed_precision_reduce.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:dtensor_reduce_scatter_lowering.mlir.test PASSED in 1.0s //tensorflow/dtensor/mlir/tests:dtensor_remove_dtensorlayout.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:dtensor_replace_auxiliary_layout_op.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:dtensor_replace_relayout_with_identity.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:dtensor_set_hlo_sharding_default.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:dtensor_xla_spmd_integration.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:elide_identity_before_copy_to_mesh.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:function_renaming.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:handle_cross_cluster_dependencies.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:handle_sparsetensors.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:layout_propagation_v2.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:lower_send_recv.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:merge_clusters.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:mesh_propagation.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:multi_device_expansion.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:op_to_device_cluster.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:propagate_default_layout.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:propagate_device_id_to_function.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:restore_and_assign.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:restore_shape_inference.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:set_default_sharding.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:sparse_expansion.mlir.test PASSED in 1.5s //tensorflow/dtensor/mlir/tests:spmd_batchparallel.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_concat.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:spmd_conv.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:spmd_einsum.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:spmd_expansion.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:spmd_io_ops.mlir.test PASSED in 0.8s //tensorflow/dtensor/mlir/tests:spmd_iterator.mlir.test PASSED in 0.9s //tensorflow/dtensor/mlir/tests:spmd_matmul.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:spmd_random.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_save_restore.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_segment_sum.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:spmd_slice.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_softmax_loss.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:spmd_squeeze.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:spmd_var_handle.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:tf_dtensor_ops.mlir.test PASSED in 0.5s //tensorflow/dtensor/mlir/tests:tpu_add_resource_device_attribute.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:tpu_integration.mlir.test PASSED in 0.7s //tensorflow/dtensor/mlir/tests:undo_merge_const_across_mesh.mlir.test PASSED in 0.6s //tensorflow/dtensor/mlir/tests:update_tpu_metadata.mlir.test PASSED in 0.6s //tensorflow/dtensor/python/tests:collective_combine_all_reduce_test_cpu PASSED in 16.6s //tensorflow/dtensor/python/tests:collective_test_cpu PASSED in 15.1s //tensorflow/dtensor/python/tests:config_test_cpu PASSED in 6.1s //tensorflow/dtensor/python/tests:device_test_cpu PASSED in 42.4s //tensorflow/dtensor/python/tests:layout_test_cpu PASSED in 6.4s //tensorflow/dtensor/python/tests:multi_client_test_cpu PASSED in 11.9s //tensorflow/dtensor/python/tests:numpy_util_test_cpu PASSED in 8.8s //tensorflow/dtensor/tests:executable_manager_test PASSED in 31.4s //tensorflow/dtensor/tests:layout_to_xla_sharding_test PASSED in 0.3s //tensorflow/dtensor/tests:tensor_layout_test PASSED in 0.2s //tensorflow/examples/adding_an_op:fact_test PASSED in 11.7s //tensorflow/examples/adding_an_op:zero_out_1_test PASSED in 11.4s //tensorflow/examples/adding_an_op:zero_out_2_test PASSED in 12.5s //tensorflow/examples/adding_an_op:zero_out_3_test PASSED in 12.7s //tensorflow/examples/custom_ops_doc/multiplex_1:multiplex_1_test PASSED in 11.0s //tensorflow/examples/custom_ops_doc/multiplex_2:multiplex_2_test_cpu PASSED in 14.1s //tensorflow/examples/custom_ops_doc/multiplex_3:multiplex_3_test PASSED in 13.2s //tensorflow/examples/custom_ops_doc/multiplex_4:multiplex_4_test PASSED in 14.6s //tensorflow/examples/custom_ops_doc/simple_hash_table:simple_hash_table_test PASSED in 13.2s //tensorflow/examples/custom_ops_doc/sleep:sleep_test PASSED in 12.7s //tensorflow/examples/speech_commands:accuracy_utils_test PASSED in 2.1s //tensorflow/examples/speech_commands:models_test PASSED in 12.3s //tensorflow/examples/speech_commands:recognize_commands_test PASSED in 2.4s //tensorflow/examples/wav_to_spectrogram:wav_to_spectrogram_test PASSED in 2.1s //tensorflow/js:ts_op_gen_test PASSED in 0.1s //tensorflow/python:array_grad_test_cpu PASSED in 8.7s //tensorflow/python:autograph_ops_test PASSED in 5.6s //tensorflow/python:batch_norm_benchmark_cpu PASSED in 6.2s //tensorflow/python:bincount_ops_test_cpu PASSED in 10.8s //tensorflow/python:bitwise_ops_test_cpu PASSED in 8.0s //tensorflow/python:clip_ops_test PASSED in 8.1s //tensorflow/python:clustering_ops_test PASSED in 21.0s //tensorflow/python:collective_ops_benchmark_cpu PASSED in 7.7s //tensorflow/python:collective_ops_gpu_test_2gpu PASSED in 8.3s //tensorflow/python:collective_ops_gpu_test_cpu PASSED in 7.8s //tensorflow/python:collective_ops_test PASSED in 17.4s //tensorflow/python:collective_ops_xla_test PASSED in 7.3s //tensorflow/python:compiled_collective_ops_gpu_test_2gpu PASSED in 6.2s //tensorflow/python:compiled_collective_ops_gpu_test_cpu PASSED in 7.1s //tensorflow/python:concat_benchmark_cpu PASSED in 6.9s //tensorflow/python:control_flow_ops_benchmark_cpu PASSED in 6.4s //tensorflow/python:control_flow_v2_enable_test PASSED in 7.9s //tensorflow/python:control_flow_v2_toggles_test PASSED in 10.2s //tensorflow/python:dequantize_op_test PASSED in 6.7s //tensorflow/python:embedding_ops_test_cpu PASSED in 8.6s //tensorflow/python:factory_ops_test_cpu PASSED in 6.8s //tensorflow/python:functional_ops_test PASSED in 6.7s //tensorflow/python:gradient_checker_v2_test_cpu PASSED in 21.1s //tensorflow/python:gradients_test_cpu PASSED in 13.0s //tensorflow/python:init_ops_test_cpu PASSED in 7.2s //tensorflow/python:init_ops_v2_test_cpu PASSED in 11.1s //tensorflow/python:math_grad_test_cpu PASSED in 16.5s //tensorflow/python:math_ops_linspace_test_cpu PASSED in 6.6s //tensorflow/python:math_ops_test_cpu PASSED in 23.5s //tensorflow/python:matmul_benchmark_cpu PASSED in 5.9s //tensorflow/python:nn_grad_test_cpu PASSED in 11.3s //tensorflow/python:nn_loss_scaling_utilities_test PASSED in 10.9s //tensorflow/python:nn_test_cpu PASSED in 54.8s //tensorflow/python:nn_xent_test_cpu PASSED in 6.5s //tensorflow/python:op_selector_test PASSED in 6.0s //tensorflow/python:ops/array_ops_test PASSED in 6.0s //tensorflow/python:quantized_conv_ops_test PASSED in 7.4s //tensorflow/python:quantized_ops_test PASSED in 7.0s //tensorflow/python:raw_ops_test_cpu PASSED in 7.1s //tensorflow/python:rnn_grad_test_cpu PASSED in 7.0s //tensorflow/python:script_ops_test PASSED in 7.2s //tensorflow/python:sort_ops_test PASSED in 9.7s //tensorflow/python:sparse_ops_test PASSED in 13.9s //tensorflow/python:split_benchmark_cpu PASSED in 7.0s //tensorflow/python:tensor_array_ops_test PASSED in 45.0s //tensorflow/python:transpose_benchmark_cpu PASSED in 7.2s //tensorflow/python:variable_spec_test PASSED in 9.3s //tensorflow/python/autograph/converters:asserts_test PASSED in 7.5s //tensorflow/python/autograph/converters:break_statements_test PASSED in 6.0s //tensorflow/python/autograph/converters:call_trees_test PASSED in 6.1s //tensorflow/python/autograph/converters:conditional_expressions_test PASSED in 5.8s //tensorflow/python/autograph/converters:continue_statements_test PASSED in 7.3s //tensorflow/python/autograph/converters:control_flow_test PASSED in 11.7s //tensorflow/python/autograph/converters:directives_test PASSED in 9.2s //tensorflow/python/autograph/converters:functions_test PASSED in 6.5s //tensorflow/python/autograph/converters:list_comprehensions_test PASSED in 8.3s //tensorflow/python/autograph/converters:lists_test PASSED in 6.8s //tensorflow/python/autograph/converters:logical_expressions_test PASSED in 6.3s //tensorflow/python/autograph/converters:return_statements_test PASSED in 7.6s //tensorflow/python/autograph/converters:slices_test PASSED in 6.2s //tensorflow/python/autograph/converters:variables_test PASSED in 6.5s //tensorflow/python/autograph/core:converter_test PASSED in 6.5s //tensorflow/python/autograph/core:function_wrappers_test PASSED in 5.9s //tensorflow/python/autograph/impl:api_test PASSED in 13.7s //tensorflow/python/autograph/impl:conversion_test PASSED in 7.6s //tensorflow/python/autograph/lang:special_functions_test PASSED in 6.4s //tensorflow/python/autograph/operators:conditional_expressions_test PASSED in 6.5s //tensorflow/python/autograph/operators:control_flow_test PASSED in 16.5s //tensorflow/python/autograph/operators:data_structures_test PASSED in 6.7s //tensorflow/python/autograph/operators:exceptions_test PASSED in 7.5s //tensorflow/python/autograph/operators:logical_test PASSED in 6.6s //tensorflow/python/autograph/operators:py_builtins_test PASSED in 13.7s //tensorflow/python/autograph/operators:slices_test PASSED in 8.7s //tensorflow/python/autograph/operators:variables_test PASSED in 6.6s //tensorflow/python/autograph/pyct:anno_test PASSED in 5.9s //tensorflow/python/autograph/pyct:ast_util_test PASSED in 6.0s //tensorflow/python/autograph/pyct:cache_test PASSED in 5.9s //tensorflow/python/autograph/pyct:cfg_test PASSED in 6.7s //tensorflow/python/autograph/pyct:error_utils_test PASSED in 5.9s //tensorflow/python/autograph/pyct:inspect_utils_test PASSED in 6.8s //tensorflow/python/autograph/pyct:loader_test PASSED in 7.1s //tensorflow/python/autograph/pyct:naming_test PASSED in 6.6s //tensorflow/python/autograph/pyct:origin_info_test PASSED in 9.8s //tensorflow/python/autograph/pyct:parser_test PASSED in 6.8s //tensorflow/python/autograph/pyct:pretty_printer_test PASSED in 6.2s //tensorflow/python/autograph/pyct:qual_names_test PASSED in 5.4s //tensorflow/python/autograph/pyct:templates_test PASSED in 36.9s //tensorflow/python/autograph/pyct:transformer_test PASSED in 6.0s //tensorflow/python/autograph/pyct:transpiler_test PASSED in 6.5s //tensorflow/python/autograph/pyct/static_analysis:activity_test PASSED in 18.3s //tensorflow/python/autograph/pyct/static_analysis:liveness_test PASSED in 20.4s //tensorflow/python/autograph/pyct/static_analysis:reaching_definitions_test PASSED in 6.3s //tensorflow/python/autograph/pyct/static_analysis:reaching_fndefs_test PASSED in 5.7s //tensorflow/python/autograph/pyct/static_analysis:type_inference_test PASSED in 6.1s //tensorflow/python/autograph/tests:assertion_test PASSED in 15.1s //tensorflow/python/autograph/tests:basic_ifexp_test PASSED in 20.6s //tensorflow/python/autograph/tests:call_to_builtin_function_test PASSED in 13.6s //tensorflow/python/autograph/tests:call_to_lambda_function_test PASSED in 12.9s //tensorflow/python/autograph/tests:call_to_named_tuple_test PASSED in 13.2s //tensorflow/python/autograph/tests:call_to_numpy_function_test PASSED in 13.6s //tensorflow/python/autograph/tests:call_to_print_function_test PASSED in 12.1s //tensorflow/python/autograph/tests:call_to_tf_api_test PASSED in 12.2s //tensorflow/python/autograph/tests:call_to_user_function_test PASSED in 13.7s //tensorflow/python/autograph/tests:composite_names_in_control_flow_test PASSED in 21.8s //tensorflow/python/autograph/tests:cond_basic_test PASSED in 21.9s //tensorflow/python/autograph/tests:datasets_test PASSED in 20.9s //tensorflow/python/autograph/tests:early_return_test PASSED in 20.7s //tensorflow/python/autograph/tests:ext_slice_test PASSED in 13.4s //tensorflow/python/autograph/tests:generator_test PASSED in 11.4s //tensorflow/python/autograph/tests:logical_expression_test PASSED in 15.9s //tensorflow/python/autograph/tests:loop_basic_test PASSED in 72.6s //tensorflow/python/autograph/tests:loop_control_flow_illegal_cases_test PASSED in 13.1s //tensorflow/python/autograph/tests:loop_created_variables_test PASSED in 20.4s //tensorflow/python/autograph/tests:loop_scoping_test PASSED in 34.1s //tensorflow/python/autograph/tests:loop_with_function_call_test PASSED in 31.3s //tensorflow/python/autograph/tests:loop_with_variable_type_illegal_cases_test PASSED in 17.3s //tensorflow/python/autograph/tests:loop_with_variable_type_test PASSED in 31.6s //tensorflow/python/autograph/tests:nested_control_flow_test PASSED in 45.6s //tensorflow/python/autograph/tests:type_annotations_test PASSED in 13.6s //tensorflow/python/autograph/utils:context_managers_test PASSED in 6.5s //tensorflow/python/autograph/utils:misc_test PASSED in 36.4s //tensorflow/python/autograph/utils:tensor_list_test PASSED in 6.2s //tensorflow/python/autograph/utils:tensors_test PASSED in 6.2s //tensorflow/python/checkpoint:benchmarks_test PASSED in 7.1s //tensorflow/python/checkpoint:checkpoint_management_test_cpu PASSED in 27.4s //tensorflow/python/checkpoint:checkpoint_metrics_test PASSED in 21.3s //tensorflow/python/checkpoint:checkpoint_test PASSED in 25.1s //tensorflow/python/checkpoint:checkpoint_view_test PASSED in 32.2s //tensorflow/python/checkpoint:checkpoint_with_v1_optimizers_test PASSED in 9.3s //tensorflow/python/checkpoint:functional_saver_test_cpu PASSED in 8.5s //tensorflow/python/checkpoint:restore_test PASSED in 41.7s //tensorflow/python/checkpoint:save_util_v1_test PASSED in 7.3s //tensorflow/python/checkpoint:saveable_compat_test PASSED in 8.5s //tensorflow/python/checkpoint:tensor_callable_test PASSED in 7.3s //tensorflow/python/checkpoint:trackable_view_test PASSED in 6.2s //tensorflow/python/client:device_lib_test_cpu PASSED in 6.4s //tensorflow/python/client:events_writer_test PASSED in 8.5s //tensorflow/python/client:session_benchmark_cpu PASSED in 7.0s //tensorflow/python/client:session_list_devices_test PASSED in 42.0s //tensorflow/python/client:session_partial_run_test PASSED in 45.2s //tensorflow/python/client:timeline_test_cpu PASSED in 38.4s //tensorflow/python/client:virtual_gpu_test_cpu PASSED in 6.7s //tensorflow/python/compat:compat_test PASSED in 15.6s //tensorflow/python/compat:disable_v2_behavior_test PASSED in 7.1s //tensorflow/python/compiler/mlir:mlir_test PASSED in 6.4s //tensorflow/python/compiler/tensorrt:trt_convert_test_cpu PASSED in 13.6s //tensorflow/python/compiler/tensorrt/test:batch_matmul_test_cpu PASSED in 34.1s //tensorflow/python/compiler/tensorrt/test:biasadd_matmul_test_cpu PASSED in 32.5s //tensorflow/python/compiler/tensorrt/test:binary_tensor_weight_broadcast_test_cpu PASSED in 7.3s //tensorflow/python/compiler/tensorrt/test:bool_test_cpu PASSED in 7.1s //tensorflow/python/compiler/tensorrt/test:cast_test_cpu PASSED in 7.6s //tensorflow/python/compiler/tensorrt/test:concatenation_test_cpu PASSED in 9.7s //tensorflow/python/compiler/tensorrt/test:const_broadcast_test_cpu PASSED in 8.2s //tensorflow/python/compiler/tensorrt/test:data_dependent_shape_test_cpu PASSED in 8.9s //tensorflow/python/compiler/tensorrt/test:dynamic_input_shapes_test_cpu PASSED in 7.3s //tensorflow/python/compiler/tensorrt/test:identity_output_test_cpu PASSED in 6.6s //tensorflow/python/compiler/tensorrt/test:int32_test_cpu PASSED in 7.6s //tensorflow/python/compiler/tensorrt/test:lru_cache_test_cpu PASSED in 36.9s //tensorflow/python/compiler/tensorrt/test:memory_alignment_test_cpu PASSED in 7.2s //tensorflow/python/compiler/tensorrt/test:multi_connection_neighbor_engine_test_cpu PASSED in 6.5s //tensorflow/python/compiler/tensorrt/test:neighboring_engine_test_cpu PASSED in 57.6s //tensorflow/python/compiler/tensorrt/test:quantization_test_cpu PASSED in 6.9s //tensorflow/python/compiler/tensorrt/test:rank_two_test_cpu PASSED in 22.0s //tensorflow/python/compiler/tensorrt/test:reshape_transpose_test_cpu PASSED in 35.6s //tensorflow/python/compiler/tensorrt/test:topk_test_cpu PASSED in 15.2s //tensorflow/python/compiler/tensorrt/test:trt_engine_op_shape_test_cpu PASSED in 39.8s //tensorflow/python/compiler/tensorrt/test:trt_mode_test_cpu PASSED in 10.7s //tensorflow/python/compiler/tensorrt/test:unary_test_cpu PASSED in 7.4s //tensorflow/python/compiler/tensorrt/test:vgg_block_nchw_test_cpu PASSED in 23.6s //tensorflow/python/compiler/tensorrt/test:vgg_block_test_cpu PASSED in 9.6s //tensorflow/python/compiler/xla:jit_compile_test_cpu PASSED in 14.0s //tensorflow/python/compiler/xla:jit_test_cpu PASSED in 17.1s //tensorflow/python/compiler/xla:xla_test_cpu PASSED in 15.6s //tensorflow/python/compiler/xla/experimental:xla_sharding_test PASSED in 62.0s //tensorflow/python/data/benchmarks:batch_benchmark PASSED in 7.1s //tensorflow/python/data/benchmarks:filter_benchmark PASSED in 7.0s //tensorflow/python/data/benchmarks:from_tensor_slices_benchmark PASSED in 7.1s //tensorflow/python/data/benchmarks:interleave_benchmark PASSED in 7.0s //tensorflow/python/data/benchmarks:list_files_benchmark PASSED in 6.3s //tensorflow/python/data/benchmarks:map_benchmark PASSED in 13.8s //tensorflow/python/data/benchmarks:meta_benchmark PASSED in 15.1s //tensorflow/python/data/benchmarks:prefetch_benchmark PASSED in 10.6s //tensorflow/python/data/benchmarks:range_benchmark PASSED in 35.6s //tensorflow/python/data/experimental/benchmarks:autotune_benchmark PASSED in 6.6s //tensorflow/python/data/experimental/benchmarks:csv_dataset_benchmark PASSED in 11.7s //tensorflow/python/data/experimental/benchmarks:map_and_batch_benchmark PASSED in 6.6s //tensorflow/python/data/experimental/benchmarks:map_defun_benchmark PASSED in 11.8s //tensorflow/python/data/experimental/benchmarks:matching_files_benchmark PASSED in 6.9s //tensorflow/python/data/experimental/benchmarks:optimize_benchmark PASSED in 10.0s //tensorflow/python/data/experimental/benchmarks:parameter_value_benchmark PASSED in 7.2s //tensorflow/python/data/experimental/benchmarks:rejection_resample_benchmark PASSED in 31.6s //tensorflow/python/data/experimental/benchmarks:snapshot_dataset_benchmark PASSED in 7.1s //tensorflow/python/data/experimental/benchmarks:unbatch_benchmark PASSED in 8.1s //tensorflow/python/data/experimental/kernel_tests:assert_cardinality_test PASSED in 108.3s //tensorflow/python/data/experimental/kernel_tests:assert_next_test PASSED in 10.0s //tensorflow/python/data/experimental/kernel_tests:assert_prev_test PASSED in 8.1s //tensorflow/python/data/experimental/kernel_tests:checkpoint_input_pipeline_hook_test PASSED in 14.1s //tensorflow/python/data/experimental/kernel_tests:compression_ops_test PASSED in 53.0s //tensorflow/python/data/experimental/kernel_tests:copy_to_device_test_cpu PASSED in 55.4s //tensorflow/python/data/experimental/kernel_tests:dense_to_sparse_batch_test PASSED in 17.8s //tensorflow/python/data/experimental/kernel_tests:from_list_test PASSED in 75.2s //tensorflow/python/data/experimental/kernel_tests:io_test PASSED in 90.3s //tensorflow/python/data/experimental/kernel_tests:lookup_ops_test PASSED in 8.9s //tensorflow/python/data/experimental/kernel_tests:make_csv_dataset_test PASSED in 37.2s //tensorflow/python/data/experimental/kernel_tests:make_saveable_from_iterator_test PASSED in 7.3s //tensorflow/python/data/experimental/kernel_tests:make_tf_record_dataset_test PASSED in 50.9s //tensorflow/python/data/experimental/kernel_tests:map_defun_op_test PASSED in 6.2s //tensorflow/python/data/experimental/kernel_tests:matching_files_dataset_test PASSED in 16.1s //tensorflow/python/data/experimental/kernel_tests:model_dataset_test PASSED in 7.5s //tensorflow/python/data/experimental/kernel_tests:non_serializable_test PASSED in 7.4s //tensorflow/python/data/experimental/kernel_tests:prefetch_to_device_test_cpu PASSED in 21.0s //tensorflow/python/data/experimental/kernel_tests:prefetch_with_slack_test PASSED in 10.4s //tensorflow/python/data/experimental/kernel_tests:shuffle_and_repeat_test PASSED in 23.8s //tensorflow/python/data/experimental/kernel_tests:sleep_test PASSED in 6.0s //tensorflow/python/data/experimental/kernel_tests:tf_record_writer_test PASSED in 11.6s //tensorflow/python/data/experimental/kernel_tests:variant_test PASSED in 7.5s //tensorflow/python/data/experimental/kernel_tests:wrap_unwrap_test_cpu PASSED in 6.4s //tensorflow/python/data/experimental/kernel_tests/optimization:filter_fusion_test PASSED in 29.0s //tensorflow/python/data/experimental/kernel_tests/optimization:filter_parallelization_test PASSED in 49.6s //tensorflow/python/data/experimental/kernel_tests/optimization:grappler_test_cpu PASSED in 8.3s //tensorflow/python/data/experimental/kernel_tests/optimization:make_deterministic_test PASSED in 24.6s //tensorflow/python/data/experimental/kernel_tests/optimization:map_and_batch_fusion_test PASSED in 7.9s //tensorflow/python/data/experimental/kernel_tests/optimization:map_and_filter_fusion_test PASSED in 19.1s //tensorflow/python/data/experimental/kernel_tests/optimization:map_fusion_test PASSED in 19.9s //tensorflow/python/data/experimental/kernel_tests/optimization:map_parallelization_test PASSED in 12.2s //tensorflow/python/data/experimental/kernel_tests/optimization:noop_elimination_test PASSED in 10.2s //tensorflow/python/data/experimental/kernel_tests/service:multi_device_test PASSED in 20.2s //tensorflow/python/data/experimental/service:server_lib_test PASSED in 7.8s //tensorflow/python/data/kernel_tests:as_numpy_iterator_test PASSED in 7.9s //tensorflow/python/data/kernel_tests:bucket_by_sequence_length_test PASSED in 17.6s //tensorflow/python/data/kernel_tests:cache_test PASSED in 46.1s //tensorflow/python/data/kernel_tests:cardinality_test PASSED in 10.7s //tensorflow/python/data/kernel_tests:checkpoint_test PASSED in 20.8s //tensorflow/python/data/kernel_tests:concatenate_test PASSED in 61.1s //tensorflow/python/data/kernel_tests:counter_test PASSED in 27.7s //tensorflow/python/data/kernel_tests:dataset_spec_test PASSED in 9.1s //tensorflow/python/data/kernel_tests:dataset_test PASSED in 20.7s //tensorflow/python/data/kernel_tests:enumerate_test PASSED in 23.5s //tensorflow/python/data/kernel_tests:from_sparse_tensor_slices_test PASSED in 7.5s //tensorflow/python/data/kernel_tests:from_tensor_slices_test PASSED in 43.5s //tensorflow/python/data/kernel_tests:from_tensors_test PASSED in 20.5s //tensorflow/python/data/kernel_tests:get_single_element_test PASSED in 8.7s //tensorflow/python/data/kernel_tests:ignore_errors_test PASSED in 14.8s //tensorflow/python/data/kernel_tests:io_test PASSED in 46.8s //tensorflow/python/data/kernel_tests:iterator_test_cpu PASSED in 16.9s //tensorflow/python/data/kernel_tests:len_test PASSED in 7.5s //tensorflow/python/data/kernel_tests:list_files_test PASSED in 9.4s //tensorflow/python/data/kernel_tests:optional_test_cpu PASSED in 30.0s //tensorflow/python/data/kernel_tests:options_test PASSED in 9.0s //tensorflow/python/data/kernel_tests:placement_test_cpu PASSED in 11.1s //tensorflow/python/data/kernel_tests:prefetch_test PASSED in 41.7s //tensorflow/python/data/kernel_tests:random_test PASSED in 23.5s //tensorflow/python/data/kernel_tests:range_test PASSED in 33.4s //tensorflow/python/data/kernel_tests:rebatch_test PASSED in 6.2s //tensorflow/python/data/kernel_tests:reduce_test_cpu PASSED in 24.5s //tensorflow/python/data/kernel_tests:scan_test_cpu PASSED in 37.8s //tensorflow/python/data/kernel_tests:sparse_batch_test PASSED in 14.9s //tensorflow/python/data/kernel_tests:unbatch_test PASSED in 29.7s //tensorflow/python/data/util:convert_test PASSED in 6.7s //tensorflow/python/data/util:nest_test PASSED in 36.0s //tensorflow/python/data/util:options_test PASSED in 6.0s //tensorflow/python/data/util:random_seed_test PASSED in 6.6s //tensorflow/python/data/util:sparse_test PASSED in 7.0s //tensorflow/python/data/util:structure_test PASSED in 7.1s //tensorflow/python/data/util:traverse_test PASSED in 5.7s //tensorflow/python/debug/cli:analyzer_cli_test_cpu PASSED in 7.8s //tensorflow/python/debug/cli:cli_config_test PASSED in 6.2s //tensorflow/python/debug/cli:cli_shared_test PASSED in 5.5s //tensorflow/python/debug/cli:command_parser_test PASSED in 6.2s //tensorflow/python/debug/cli:curses_ui_test PASSED in 6.1s //tensorflow/python/debug/cli:debugger_cli_common_test PASSED in 6.4s //tensorflow/python/debug/cli:evaluator_test PASSED in 5.9s //tensorflow/python/debug/cli:profile_analyzer_cli_test PASSED in 5.7s //tensorflow/python/debug/cli:readline_ui_test PASSED in 5.6s //tensorflow/python/debug/cli:tensor_format_test PASSED in 6.5s //tensorflow/python/debug/lib:check_numerics_callback_test_cpu PASSED in 10.7s //tensorflow/python/debug/lib:common_test PASSED in 5.7s //tensorflow/python/debug/lib:debug_data_test PASSED in 6.5s //tensorflow/python/debug/lib:debug_events_monitors_test PASSED in 6.5s //tensorflow/python/debug/lib:debug_events_writer_test PASSED in 7.2s //tensorflow/python/debug/lib:debug_gradients_test_cpu PASSED in 6.0s //tensorflow/python/debug/lib:debug_graph_reconstruction_test_cpu PASSED in 6.6s //tensorflow/python/debug/lib:debug_graphs_test PASSED in 7.7s //tensorflow/python/debug/lib:debug_grappler_test_cpu PASSED in 6.5s //tensorflow/python/debug/lib:debug_utils_test PASSED in 6.4s //tensorflow/python/debug/lib:debug_v2_ops_test_cpu PASSED in 15.9s //tensorflow/python/debug/lib:profiling_test PASSED in 6.0s //tensorflow/python/debug/lib:session_debug_file_test_cpu PASSED in 9.7s //tensorflow/python/debug/lib:session_debug_multi_gpu_test_cpu PASSED in 6.1s //tensorflow/python/debug/lib:source_utils_test PASSED in 8.4s //tensorflow/python/debug/wrappers:disk_usage_test PASSED in 6.2s //tensorflow/python/debug/wrappers:dumping_wrapper_test PASSED in 6.3s //tensorflow/python/debug/wrappers:framework_test PASSED in 8.2s //tensorflow/python/debug/wrappers:local_cli_wrapper_test PASSED in 6.8s //tensorflow/python/distribute:checkpoint_utils_test_2gpu PASSED in 11.0s //tensorflow/python/distribute:checkpoint_utils_test_cpu PASSED in 8.7s //tensorflow/python/distribute:checkpointing_test_2gpu PASSED in 6.9s //tensorflow/python/distribute:checkpointing_test_cpu PASSED in 7.4s //tensorflow/python/distribute:collective_all_reduce_strategy_test_2gpu PASSED in 56.5s //tensorflow/python/distribute:collective_all_reduce_strategy_test_cpu PASSED in 61.3s //tensorflow/python/distribute:collective_all_reduce_strategy_test_xla_2gpu PASSED in 25.6s //tensorflow/python/distribute:collective_util_test PASSED in 5.8s //tensorflow/python/distribute:combinations_test_2gpu PASSED in 18.2s //tensorflow/python/distribute:combinations_test_cpu PASSED in 17.7s //tensorflow/python/distribute:cross_device_utils_test_cpu PASSED in 8.2s //tensorflow/python/distribute:custom_training_loop_gradient_test_2gpu PASSED in 18.2s //tensorflow/python/distribute:custom_training_loop_gradient_test_cpu PASSED in 9.7s //tensorflow/python/distribute:device_util_test_cpu PASSED in 12.5s //tensorflow/python/distribute:distribute_coordinator_test PASSED in 18.1s //tensorflow/python/distribute:distribute_lib_test PASSED in 9.5s //tensorflow/python/distribute:distribute_utils_test_2gpu PASSED in 52.7s //tensorflow/python/distribute:distribute_utils_test_cpu PASSED in 12.7s //tensorflow/python/distribute:input_ops_test_cpu PASSED in 13.1s //tensorflow/python/distribute:metrics_v1_test_2gpu PASSED in 26.3s //tensorflow/python/distribute:metrics_v1_test_cpu PASSED in 25.6s //tensorflow/python/distribute:mirrored_values_test_2gpu PASSED in 8.0s //tensorflow/python/distribute:mirrored_values_test_cpu PASSED in 7.4s //tensorflow/python/distribute:mirrored_variable_test_2gpu PASSED in 16.2s //tensorflow/python/distribute:mirrored_variable_test_cpu PASSED in 72.2s //tensorflow/python/distribute:multi_process_runner_no_init_test PASSED in 6.5s //tensorflow/python/distribute:multi_worker_continuous_run_test_cpu PASSED in 18.6s //tensorflow/python/distribute:multi_worker_util_test PASSED in 6.4s //tensorflow/python/distribute:numpy_dataset_test PASSED in 8.1s //tensorflow/python/distribute:one_device_strategy_test_cpu PASSED in 62.0s //tensorflow/python/distribute:packed_distributed_variable_test PASSED in 6.9s //tensorflow/python/distribute:parameter_server_strategy_test_2gpu PASSED in 22.9s //tensorflow/python/distribute:parameter_server_strategy_test_cpu PASSED in 22.1s //tensorflow/python/distribute:parameter_server_strategy_v2_test_2gpu PASSED in 21.8s //tensorflow/python/distribute:parameter_server_strategy_v2_test_cpu PASSED in 20.8s //tensorflow/python/distribute:per_replica_test_2gpu PASSED in 9.5s //tensorflow/python/distribute:per_replica_test_cpu PASSED in 9.3s //tensorflow/python/distribute:ps_values_test_2gpu PASSED in 9.8s //tensorflow/python/distribute:ps_values_test_cpu PASSED in 10.5s //tensorflow/python/distribute:remote_mirrored_strategy_eager_test_cpu PASSED in 7.7s //tensorflow/python/distribute:sharded_variable_test PASSED in 19.3s //tensorflow/python/distribute:shared_variable_creator_test PASSED in 6.6s //tensorflow/python/distribute:strategy_combinations_test_cpu PASSED in 41.8s //tensorflow/python/distribute:template_mirrored_strategy_test_cpu PASSED in 6.8s //tensorflow/python/distribute:test_util_test_2gpu PASSED in 66.6s //tensorflow/python/distribute:test_util_test_cpu PASSED in 13.8s //tensorflow/python/distribute:tf_function_test_2gpu PASSED in 7.2s //tensorflow/python/distribute:tf_function_test_cpu PASSED in 10.4s //tensorflow/python/distribute:values_v2_test_cpu PASSED in 10.1s //tensorflow/python/distribute:warm_starting_util_test_2gpu PASSED in 11.4s //tensorflow/python/distribute:warm_starting_util_test_cpu PASSED in 8.0s //tensorflow/python/distribute/cluster_resolver:base_cluster_resolver_py_test PASSED in 6.2s //tensorflow/python/distribute/cluster_resolver:gce_cluster_resolver_py_test PASSED in 6.1s //tensorflow/python/distribute/cluster_resolver:kubernetes_cluster_resolver_py_test PASSED in 43.5s //tensorflow/python/distribute/cluster_resolver:sagemaker_cluster_resolver_py_test PASSED in 8.6s //tensorflow/python/distribute/cluster_resolver:slurm_cluster_resolver_py_test PASSED in 6.7s //tensorflow/python/distribute/cluster_resolver:tfconfig_cluster_resolver_py_test PASSED in 6.6s //tensorflow/python/distribute/cluster_resolver/tpu:tpu_cluster_resolver_py_test PASSED in 8.4s //tensorflow/python/distribute/coordinator:metric_utils_test PASSED in 7.6s //tensorflow/python/distribute/coordinator:watchdog_test PASSED in 60.1s //tensorflow/python/distribute/experimental:dtensor_util_test_cpu PASSED in 12.4s //tensorflow/python/distribute/experimental:mirrored_strategy_test_cpu PASSED in 26.3s //tensorflow/python/distribute/integration_test:saved_model_test_cpu PASSED in 34.9s //tensorflow/python/distribute/parallel_device:parallel_device_test_cpu PASSED in 15.4s //tensorflow/python/distribute/v1:all_reduce_test PASSED in 39.5s //tensorflow/python/distribute/v1:cross_device_ops_test_2gpu PASSED in 58.0s //tensorflow/python/distribute/v1:cross_device_ops_test_cpu PASSED in 42.6s //tensorflow/python/dlpack:dlpack_test_cpu PASSED in 8.2s //tensorflow/python/eager:backprop_test_cpu PASSED in 107.5s //tensorflow/python/eager:benchmarks_test_cpu PASSED in 5.7s //tensorflow/python/eager:cancellation_test_cpu PASSED in 6.1s //tensorflow/python/eager:context_test_cpu PASSED in 8.0s //tensorflow/python/eager:core_test_cpu PASSED in 15.7s //tensorflow/python/eager:gradient_input_output_exclusions_test PASSED in 30.3s //tensorflow/python/eager:graph_only_ops_test_cpu PASSED in 6.1s //tensorflow/python/eager:lift_to_graph_test PASSED in 7.8s //tensorflow/python/eager:monitoring_test_cpu PASSED in 8.3s //tensorflow/python/eager:ops_test_cpu PASSED in 13.6s //tensorflow/python/eager:profiler_client_test PASSED in 7.2s //tensorflow/python/eager:profiler_test_cpu PASSED in 42.1s //tensorflow/python/eager:pywrap_tfe_test PASSED in 17.1s //tensorflow/python/eager:record_test PASSED in 9.7s //tensorflow/python/eager:remote_benchmarks_test_cpu PASSED in 6.8s //tensorflow/python/eager:run_eager_op_as_function_test_cpu PASSED in 25.3s //tensorflow/python/eager:run_eager_op_as_function_xla_test_cpu PASSED in 6.1s //tensorflow/python/eager:small_constants_optimizer_test_cpu PASSED in 186.6s //tensorflow/python/eager:tensor_test_cpu PASSED in 11.6s //tensorflow/python/eager:wrap_function_device_test_cpu PASSED in 6.9s //tensorflow/python/eager:wrap_function_test PASSED in 8.1s //tensorflow/python/eager/benchmarks:kpi_benchmark_test_cpu PASSED in 13.6s //tensorflow/python/eager/memory_tests:remote_memory_test_cpu PASSED in 7.6s //tensorflow/python/eager/polymorphic_function:argument_naming_test_cpu PASSED in 9.9s //tensorflow/python/eager/polymorphic_function:collection_test_cpu PASSED in 6.2s //tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu PASSED in 7.6s //tensorflow/python/eager/polymorphic_function:compiler_ir_test_cpu_mlir_bridge_test PASSED in 7.4s //tensorflow/python/eager/polymorphic_function:function_spec_test PASSED in 6.3s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_jit_test_cpu PASSED in 24.5s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_jit_test_cpu_mlir_bridge_test PASSED in 44.0s //tensorflow/python/eager/polymorphic_function:polymorphic_function_xla_test_cpu PASSED in 7.3s //tensorflow/python/eager/polymorphic_function:quarantine_test PASSED in 23.6s //tensorflow/python/feature_column:sequence_feature_column_integration_test PASSED in 7.5s //tensorflow/python/feature_column:serialization_test PASSED in 11.3s //tensorflow/python/framework:auto_control_deps_test PASSED in 20.0s //tensorflow/python/framework:c_api_util_test PASSED in 8.4s //tensorflow/python/framework:common_shapes_test PASSED in 6.6s //tensorflow/python/framework:composite_tensor_test PASSED in 9.5s //tensorflow/python/framework:config_test_2gpu PASSED in 10.7s //tensorflow/python/framework:config_test_cpu PASSED in 9.8s //tensorflow/python/framework:constant_op_test PASSED in 10.2s //tensorflow/python/framework:device_spec_test PASSED in 6.0s //tensorflow/python/framework:device_test PASSED in 29.0s //tensorflow/python/framework:dtypes_test PASSED in 11.7s //tensorflow/python/framework:error_interpolation_test PASSED in 7.6s //tensorflow/python/framework:errors_test PASSED in 7.3s //tensorflow/python/framework:extension_type_field_test PASSED in 7.5s //tensorflow/python/framework:extension_type_test PASSED in 17.0s //tensorflow/python/framework:file_system_test PASSED in 7.3s //tensorflow/python/framework:function_def_to_graph_test PASSED in 8.6s //tensorflow/python/framework:graph_building_benchmark_cpu PASSED in 6.3s //tensorflow/python/framework:graph_util_test PASSED in 7.0s //tensorflow/python/framework:immutable_dict_test PASSED in 6.1s //tensorflow/python/framework:importer_test PASSED in 8.4s //tensorflow/python/framework:indexed_slices_test PASSED in 6.2s //tensorflow/python/framework:kernels_test PASSED in 7.6s //tensorflow/python/framework:meta_graph_test PASSED in 7.4s //tensorflow/python/framework:node_file_writer_test_cpu PASSED in 7.3s //tensorflow/python/framework:offset_counter_helper_test PASSED in 0.2s //tensorflow/python/framework:op_allowlist_namespace_test PASSED in 1.9s //tensorflow/python/framework:op_callbacks_test_cpu PASSED in 13.4s //tensorflow/python/framework:op_def_library_test PASSED in 7.0s //tensorflow/python/framework:op_def_util_test PASSED in 6.8s //tensorflow/python/framework:ops_enable_eager_test PASSED in 1.8s //tensorflow/python/framework:ops_test PASSED in 21.2s //tensorflow/python/framework:proto_test PASSED in 7.8s //tensorflow/python/framework:py_context_manager_test PASSED in 6.2s //tensorflow/python/framework:python_api_dispatcher_test PASSED in 6.5s //tensorflow/python/framework:python_api_info_test PASSED in 7.1s //tensorflow/python/framework:python_api_parameter_converter_test PASSED in 9.0s //tensorflow/python/framework:python_op_gen_annotation_test PASSED in 3.4s //tensorflow/python/framework:python_op_gen_annotator_test PASSED in 0.3s //tensorflow/python/framework:python_tensor_converter_test PASSED in 6.5s //tensorflow/python/framework:random_seed_test PASSED in 8.8s //tensorflow/python/framework:registry_test PASSED in 6.4s //tensorflow/python/framework:smart_cond_test PASSED in 6.8s //tensorflow/python/framework:sparse_tensor_test PASSED in 7.7s //tensorflow/python/framework:subscribe_test PASSED in 6.7s //tensorflow/python/framework:tensor_shape_test PASSED in 7.6s //tensorflow/python/framework:tensor_test PASSED in 6.4s //tensorflow/python/framework:tensor_util_test PASSED in 8.6s //tensorflow/python/framework:test_combinations_test PASSED in 5.8s //tensorflow/python/framework:test_util_test_cpu PASSED in 14.1s //tensorflow/python/framework:tf2_test PASSED in 7.4s //tensorflow/python/framework:traceable_stack_test PASSED in 6.7s //tensorflow/python/framework:type_spec_test PASSED in 6.4s //tensorflow/python/framework:versions_test PASSED in 7.5s //tensorflow/python/framework/experimental:graph_building_test_cpu PASSED in 5.6s //tensorflow/python/framework/experimental:unified_api_test_cpu PASSED in 11.2s //tensorflow/python/grappler:arithmetic_optimizer_test_cpu PASSED in 7.4s //tensorflow/python/grappler:auto_mixed_precision_test_cpu PASSED in 9.7s //tensorflow/python/grappler:constant_folding_test_cpu PASSED in 6.2s //tensorflow/python/grappler:cost_analyzer_test PASSED in 44.9s //tensorflow/python/grappler:datasets_test PASSED in 7.7s //tensorflow/python/grappler:item_test PASSED in 42.7s //tensorflow/python/grappler:memory_optimizer_test PASSED in 12.6s //tensorflow/python/grappler:model_analyzer_test PASSED in 5.9s //tensorflow/python/grappler:remapper_test_cpu PASSED in 6.4s //tensorflow/python/grappler:tf_optimizer_test PASSED in 9.0s //tensorflow/python/kernel_tests:benchmark_test_cpu PASSED in 7.6s //tensorflow/python/kernel_tests:check_ops_test_cpu PASSED in 14.7s //tensorflow/python/kernel_tests:collective_ops_multi_worker_test PASSED in 28.2s //tensorflow/python/kernel_tests:composite_tensor_ops_test PASSED in 6.8s //tensorflow/python/kernel_tests:critical_section_test_cpu PASSED in 17.5s //tensorflow/python/kernel_tests:garbage_collection_test PASSED in 7.0s //tensorflow/python/kernel_tests:gradient_correctness_test_cpu PASSED in 6.6s //tensorflow/python/kernel_tests:histogram_ops_test_cpu PASSED in 8.1s //tensorflow/python/kernel_tests:logging_ops_test_cpu PASSED in 8.4s //tensorflow/python/kernel_tests:numerics_test_cpu PASSED in 7.4s //tensorflow/python/kernel_tests:template_test PASSED in 9.7s //tensorflow/python/kernel_tests:trace_op_test_cpu PASSED in 6.4s //tensorflow/python/kernel_tests/array_ops:batch_gather_op_test_cpu PASSED in 8.8s //tensorflow/python/kernel_tests/array_ops:batch_scatter_ops_test PASSED in 7.4s //tensorflow/python/kernel_tests/array_ops:batchtospace_op_test_cpu PASSED in 13.4s //tensorflow/python/kernel_tests/array_ops:bcast_ops_test PASSED in 7.3s //tensorflow/python/kernel_tests/array_ops:bitcast_op_test_cpu PASSED in 7.1s //tensorflow/python/kernel_tests/array_ops:broadcast_to_ops_test_cpu PASSED in 24.3s //tensorflow/python/kernel_tests/array_ops:cast_op_test_cpu PASSED in 10.1s //tensorflow/python/kernel_tests/array_ops:constant_op_eager_test_cpu PASSED in 7.6s //tensorflow/python/kernel_tests/array_ops:constant_op_test_cpu PASSED in 9.8s //tensorflow/python/kernel_tests/array_ops:denormal_test_cpu PASSED in 6.6s //tensorflow/python/kernel_tests/array_ops:depthtospace_op_test_cpu PASSED in 8.4s //tensorflow/python/kernel_tests/array_ops:edit_distance_op_test PASSED in 8.9s //tensorflow/python/kernel_tests/array_ops:fingerprint_op_test PASSED in 7.4s //tensorflow/python/kernel_tests/array_ops:gather_nd_op_test_cpu PASSED in 8.0s //tensorflow/python/kernel_tests/array_ops:identity_n_op_py_test PASSED in 7.3s //tensorflow/python/kernel_tests/array_ops:identity_op_py_test PASSED in 6.7s //tensorflow/python/kernel_tests/array_ops:large_concat_op_test_cpu PASSED in 7.5s //tensorflow/python/kernel_tests/array_ops:manip_ops_test_cpu PASSED in 8.0s //tensorflow/python/kernel_tests/array_ops:one_hot_op_test_cpu PASSED in 7.3s //tensorflow/python/kernel_tests/array_ops:pad_op_test_cpu PASSED in 12.4s //tensorflow/python/kernel_tests/array_ops:reshape_op_test_cpu PASSED in 8.2s //tensorflow/python/kernel_tests/array_ops:reverse_sequence_op_test_cpu PASSED in 7.9s //tensorflow/python/kernel_tests/array_ops:scalar_test_cpu PASSED in 8.0s //tensorflow/python/kernel_tests/array_ops:shape_ops_test_cpu PASSED in 15.0s //tensorflow/python/kernel_tests/array_ops:slice_op_test_cpu PASSED in 8.7s //tensorflow/python/kernel_tests/array_ops:spacetobatch_op_test_cpu PASSED in 33.9s //tensorflow/python/kernel_tests/array_ops:spacetodepth_op_test_cpu PASSED in 7.3s //tensorflow/python/kernel_tests/array_ops:stack_op_test_cpu PASSED in 12.3s //tensorflow/python/kernel_tests/array_ops:unique_op_test_cpu PASSED in 7.3s //tensorflow/python/kernel_tests/array_ops:unstack_op_test_cpu PASSED in 8.5s //tensorflow/python/kernel_tests/array_ops:where_op_test_cpu PASSED in 14.6s //tensorflow/python/kernel_tests/control_flow:cond_v2_test_cpu PASSED in 50.3s //tensorflow/python/kernel_tests/control_flow:control_flow_util_test PASSED in 9.2s //tensorflow/python/kernel_tests/control_flow:control_flow_util_v2_test PASSED in 6.9s //tensorflow/python/kernel_tests/control_flow:py_func_test_cpu PASSED in 23.2s //tensorflow/python/kernel_tests/control_flow:scan_ops_test_cpu PASSED in 69.5s //tensorflow/python/kernel_tests/control_flow:while_v2_test_cpu PASSED in 53.4s //tensorflow/python/kernel_tests/custom_ops:ackermann_test PASSED in 38.7s //tensorflow/python/kernel_tests/custom_ops:duplicate_op_test PASSED in 6.4s //tensorflow/python/kernel_tests/custom_ops:invalid_op_test PASSED in 6.1s //tensorflow/python/kernel_tests/data_structures:conditional_accumulator_test PASSED in 10.6s //tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_2gpu PASSED in 11.9s //tensorflow/python/kernel_tests/data_structures:dynamic_partition_op_test_cpu PASSED in 13.8s //tensorflow/python/kernel_tests/data_structures:dynamic_stitch_op_test_cpu PASSED in 7.2s //tensorflow/python/kernel_tests/data_structures:fifo_queue_test PASSED in 8.7s //tensorflow/python/kernel_tests/data_structures:list_ops_test_cpu PASSED in 20.0s //tensorflow/python/kernel_tests/data_structures:listdiff_op_test PASSED in 9.8s //tensorflow/python/kernel_tests/data_structures:lookup_ops_test PASSED in 20.1s //tensorflow/python/kernel_tests/data_structures:map_ops_test PASSED in 13.2s //tensorflow/python/kernel_tests/data_structures:padding_fifo_queue_test_cpu PASSED in 6.8s //tensorflow/python/kernel_tests/data_structures:priority_queue_test PASSED in 6.3s //tensorflow/python/kernel_tests/data_structures:stack_ops_test_cpu PASSED in 9.4s //tensorflow/python/kernel_tests/data_structures:stage_op_test_cpu PASSED in 9.2s //tensorflow/python/kernel_tests/distributions:bernoulli_test_cpu PASSED in 47.7s //tensorflow/python/kernel_tests/distributions:bijector_test_cpu PASSED in 7.9s //tensorflow/python/kernel_tests/distributions:categorical_test_cpu PASSED in 10.5s //tensorflow/python/kernel_tests/distributions:dirichlet_multinomial_test_cpu PASSED in 11.8s //tensorflow/python/kernel_tests/distributions:dirichlet_test_cpu PASSED in 10.8s //tensorflow/python/kernel_tests/distributions:exponential_test_cpu PASSED in 8.5s //tensorflow/python/kernel_tests/distributions:gamma_test_cpu PASSED in 42.7s //tensorflow/python/kernel_tests/distributions:identity_bijector_test_cpu PASSED in 6.7s //tensorflow/python/kernel_tests/distributions:kullback_leibler_test_cpu PASSED in 6.6s //tensorflow/python/kernel_tests/distributions:laplace_test_cpu PASSED in 26.6s //tensorflow/python/kernel_tests/distributions:multinomial_test_cpu PASSED in 7.8s //tensorflow/python/kernel_tests/distributions:normal_test_cpu PASSED in 20.7s //tensorflow/python/kernel_tests/distributions:special_math_test_cpu PASSED in 17.7s //tensorflow/python/kernel_tests/distributions:uniform_test_cpu PASSED in 9.8s //tensorflow/python/kernel_tests/image_ops:attention_ops_test PASSED in 7.0s //tensorflow/python/kernel_tests/image_ops:decode_bmp_op_test PASSED in 6.1s //tensorflow/python/kernel_tests/image_ops:decode_compressed_op_test PASSED in 6.3s //tensorflow/python/kernel_tests/image_ops:decode_image_op_test PASSED in 6.9s //tensorflow/python/kernel_tests/image_ops:decode_jpeg_op_test PASSED in 6.8s //tensorflow/python/kernel_tests/image_ops:decode_png_op_test PASSED in 8.6s //tensorflow/python/kernel_tests/image_ops:decode_raw_op_test PASSED in 7.1s //tensorflow/python/kernel_tests/image_ops:draw_bounding_box_op_test_cpu PASSED in 6.8s //tensorflow/python/kernel_tests/image_ops:extract_image_patches_op_test_cpu PASSED in 7.3s //tensorflow/python/kernel_tests/image_ops:extract_volume_patches_op_test_cpu PASSED in 6.1s //tensorflow/python/kernel_tests/io_ops:checkpoint_ops_test PASSED in 9.6s //tensorflow/python/kernel_tests/io_ops:decode_csv_op_test PASSED in 6.6s //tensorflow/python/kernel_tests/io_ops:io_ops_test PASSED in 7.6s //tensorflow/python/kernel_tests/io_ops:parse_single_example_op_test PASSED in 8.1s //tensorflow/python/kernel_tests/io_ops:parsing_ops_test PASSED in 22.4s //tensorflow/python/kernel_tests/io_ops:reader_ops_test PASSED in 9.4s //tensorflow/python/kernel_tests/io_ops:record_input_test PASSED in 23.8s //tensorflow/python/kernel_tests/io_ops:save_restore_ops_test PASSED in 7.0s //tensorflow/python/kernel_tests/linalg:determinant_op_test_cpu PASSED in 6.2s //tensorflow/python/kernel_tests/linalg:linear_operator_addition_test_cpu PASSED in 8.7s //tensorflow/python/kernel_tests/linalg:linear_operator_algebra_test_cpu PASSED in 7.2s //tensorflow/python/kernel_tests/linalg:linear_operator_test_cpu PASSED in 7.8s //tensorflow/python/kernel_tests/linalg:lu_op_test_cpu PASSED in 9.9s //tensorflow/python/kernel_tests/linalg:matrix_inverse_op_test_cpu PASSED in 8.7s //tensorflow/python/kernel_tests/linalg:matrix_logarithm_op_test PASSED in 67.0s //tensorflow/python/kernel_tests/linalg:matrix_solve_ls_op_test_cpu PASSED in 13.4s //tensorflow/python/kernel_tests/linalg:matrix_solve_op_test_cpu PASSED in 15.4s //tensorflow/python/kernel_tests/linalg:matrix_square_root_op_test_cpu PASSED in 6.0s //tensorflow/python/kernel_tests/linalg:slicing_test_cpu PASSED in 10.2s //tensorflow/python/kernel_tests/linalg/sparse:conjugate_gradient_test_cpu PASSED in 11.1s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_test_cpu PASSED in 7.6s //tensorflow/python/kernel_tests/math_ops:aggregate_ops_test_cpu PASSED in 12.5s //tensorflow/python/kernel_tests/math_ops:argmax_op_test_cpu PASSED in 8.4s //tensorflow/python/kernel_tests/math_ops:banded_triangular_solve_op_test_cpu PASSED in 9.5s //tensorflow/python/kernel_tests/math_ops:basic_gpu_test_cpu PASSED in 9.2s //tensorflow/python/kernel_tests/math_ops:bincount_op_test_cpu PASSED in 9.8s //tensorflow/python/kernel_tests/math_ops:bucketize_op_test_cpu PASSED in 9.6s //tensorflow/python/kernel_tests/math_ops:clip_ops_test PASSED in 9.7s //tensorflow/python/kernel_tests/math_ops:confusion_matrix_test PASSED in 9.1s //tensorflow/python/kernel_tests/math_ops:cross_grad_test_cpu PASSED in 7.6s //tensorflow/python/kernel_tests/math_ops:cumulative_logsumexp_test_cpu PASSED in 26.8s //tensorflow/python/kernel_tests/math_ops:in_topk_op_test_cpu PASSED in 6.1s //tensorflow/python/kernel_tests/math_ops:reduce_benchmark_test_cpu PASSED in 7.0s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_d9m_test_cpu PASSED in 7.0s //tensorflow/python/kernel_tests/math_ops:sets_test PASSED in 24.2s //tensorflow/python/kernel_tests/math_ops:topk_op_test_cpu PASSED in 7.8s //tensorflow/python/kernel_tests/math_ops:zero_division_test_cpu PASSED in 6.4s //tensorflow/python/kernel_tests/nn_ops:betainc_op_test_cpu PASSED in 9.2s //tensorflow/python/kernel_tests/nn_ops:bias_op_test_cpu PASSED in 136.8s //tensorflow/python/kernel_tests/nn_ops:conv1d_test_cpu PASSED in 7.7s //tensorflow/python/kernel_tests/nn_ops:conv1d_transpose_test_cpu PASSED in 6.9s //tensorflow/python/kernel_tests/nn_ops:conv2d_transpose_test_cpu PASSED in 6.6s //tensorflow/python/kernel_tests/nn_ops:conv3d_backprop_filter_v2_grad_test_cpu PASSED in 20.4s //tensorflow/python/kernel_tests/nn_ops:conv3d_transpose_test_cpu PASSED in 7.5s //tensorflow/python/kernel_tests/nn_ops:ctc_decoder_ops_test PASSED in 6.4s //tensorflow/python/kernel_tests/nn_ops:ctc_loss_op_test_cpu PASSED in 57.8s //tensorflow/python/kernel_tests/nn_ops:cudnn_d9m_test_cpu PASSED in 7.4s //tensorflow/python/kernel_tests/nn_ops:cudnn_deterministic_ops_test_cpu PASSED in 5.9s //tensorflow/python/kernel_tests/nn_ops:losses_test PASSED in 27.2s //tensorflow/python/kernel_tests/nn_ops:lrn_op_test_cpu PASSED in 9.2s //tensorflow/python/kernel_tests/nn_ops:morphological_ops_test_cpu PASSED in 10.3s //tensorflow/python/kernel_tests/nn_ops:nth_element_op_test_cpu PASSED in 7.4s //tensorflow/python/kernel_tests/nn_ops:pool_test_cpu PASSED in 37.4s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_3d_test_cpu PASSED in 24.9s //tensorflow/python/kernel_tests/nn_ops:relu_op_test_cpu PASSED in 9.7s //tensorflow/python/kernel_tests/nn_ops:softmax_op_test_cpu PASSED in 7.5s //tensorflow/python/kernel_tests/nn_ops:softplus_op_test_cpu PASSED in 6.1s //tensorflow/python/kernel_tests/nn_ops:softsign_op_test_cpu PASSED in 6.4s //tensorflow/python/kernel_tests/nn_ops:xent_op_d9m_test_cpu PASSED in 140.0s //tensorflow/python/kernel_tests/nn_ops:xent_op_test_cpu PASSED in 7.8s //tensorflow/python/kernel_tests/proto:descriptor_source_test PASSED in 7.5s //tensorflow/python/kernel_tests/proto:encode_proto_op_test PASSED in 8.5s //tensorflow/python/kernel_tests/quantization_ops:quantization_ops_test PASSED in 8.3s //tensorflow/python/kernel_tests/random:candidate_sampler_ops_test PASSED in 7.5s //tensorflow/python/kernel_tests/random:multinomial_op_test_cpu PASSED in 8.9s //tensorflow/python/kernel_tests/random:parameterized_truncated_normal_op_test_cpu PASSED in 11.5s //tensorflow/python/kernel_tests/random:random_crop_test_cpu PASSED in 8.0s //tensorflow/python/kernel_tests/random:random_grad_test_cpu PASSED in 9.9s //tensorflow/python/kernel_tests/random:random_ops_test_cpu PASSED in 12.0s //tensorflow/python/kernel_tests/random:random_poisson_test_cpu PASSED in 9.5s //tensorflow/python/kernel_tests/random:random_shuffle_queue_test PASSED in 6.8s //tensorflow/python/kernel_tests/random:stateful_random_ops_test_cpu PASSED in 14.2s //tensorflow/python/kernel_tests/signal:mel_ops_test_cpu PASSED in 12.1s //tensorflow/python/kernel_tests/signal:mfcc_ops_test_cpu PASSED in 6.6s //tensorflow/python/kernel_tests/signal:reconstruction_ops_test_cpu PASSED in 10.0s //tensorflow/python/kernel_tests/signal:shape_ops_test_cpu PASSED in 17.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_add_op_test PASSED in 10.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_concat_op_test PASSED in 8.5s //tensorflow/python/kernel_tests/sparse_ops:sparse_conditional_accumulator_test PASSED in 6.5s //tensorflow/python/kernel_tests/sparse_ops:sparse_cross_op_test PASSED in 10.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_matmul_op_test_cpu PASSED in 32.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_reorder_op_test PASSED in 8.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_reshape_op_test PASSED in 8.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_serialization_ops_test PASSED in 7.9s //tensorflow/python/kernel_tests/sparse_ops:sparse_slice_op_test PASSED in 9.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_split_op_test_cpu PASSED in 7.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_grad_test_cpu PASSED in 16.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_d9m_test_cpu PASSED in 33.4s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensor_dense_matmul_op_test_cpu PASSED in 28.9s //tensorflow/python/kernel_tests/sparse_ops:sparse_tensors_map_ops_test PASSED in 7.0s //tensorflow/python/kernel_tests/sparse_ops:sparse_to_dense_op_py_test_cpu PASSED in 6.6s //tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_d9m_test_cpu PASSED in 74.3s //tensorflow/python/kernel_tests/sparse_ops:sparse_xent_op_test_cpu PASSED in 8.7s //tensorflow/python/kernel_tests/sparse_ops:sparsemask_op_test PASSED in 8.6s //tensorflow/python/kernel_tests/strings_ops:as_string_op_test PASSED in 8.6s //tensorflow/python/kernel_tests/strings_ops:base64_ops_test PASSED in 12.5s //tensorflow/python/kernel_tests/strings_ops:reduce_join_op_test_cpu PASSED in 7.4s //tensorflow/python/kernel_tests/strings_ops:regex_full_match_op_test PASSED in 7.5s //tensorflow/python/kernel_tests/strings_ops:regex_replace_op_test PASSED in 7.0s //tensorflow/python/kernel_tests/strings_ops:string_bytes_split_op_test PASSED in 6.6s //tensorflow/python/kernel_tests/strings_ops:string_format_op_test PASSED in 8.2s //tensorflow/python/kernel_tests/strings_ops:string_join_op_test PASSED in 6.9s //tensorflow/python/kernel_tests/strings_ops:string_length_op_test PASSED in 6.8s //tensorflow/python/kernel_tests/strings_ops:string_lower_op_test PASSED in 6.4s //tensorflow/python/kernel_tests/strings_ops:string_split_op_test PASSED in 10.6s //tensorflow/python/kernel_tests/strings_ops:string_strip_op_test PASSED in 7.0s //tensorflow/python/kernel_tests/strings_ops:string_to_hash_bucket_op_test_cpu PASSED in 6.1s //tensorflow/python/kernel_tests/strings_ops:string_to_number_op_test_cpu PASSED in 7.4s //tensorflow/python/kernel_tests/strings_ops:string_upper_op_test PASSED in 6.6s //tensorflow/python/kernel_tests/strings_ops:substr_op_test PASSED in 9.0s //tensorflow/python/kernel_tests/strings_ops:unicode_decode_op_test PASSED in 14.5s //tensorflow/python/kernel_tests/strings_ops:unicode_encode_op_test PASSED in 8.3s //tensorflow/python/kernel_tests/strings_ops:unicode_script_op_test PASSED in 6.6s //tensorflow/python/kernel_tests/strings_ops:unicode_transcode_op_test PASSED in 9.0s //tensorflow/python/kernel_tests/strings_ops:unsorted_segment_join_op_test_cpu PASSED in 7.5s //tensorflow/python/kernel_tests/summary_ops:summary_ops_test_cpu PASSED in 14.9s //tensorflow/python/kernel_tests/summary_ops:summary_v1_audio_op_test_cpu PASSED in 6.4s //tensorflow/python/kernel_tests/summary_ops:summary_v1_image_op_test_cpu PASSED in 8.2s //tensorflow/python/kernel_tests/summary_ops:summary_v1_ops_test PASSED in 6.8s //tensorflow/python/kernel_tests/summary_ops:summary_v1_tensor_op_test PASSED in 6.9s //tensorflow/python/kernel_tests/v1_compat_tests:array_ops_test_cpu PASSED in 7.3s //tensorflow/python/kernel_tests/v1_compat_tests:dense_update_ops_test_cpu PASSED in 5.9s //tensorflow/python/kernel_tests/v1_compat_tests:identity_op_py_test PASSED in 6.8s //tensorflow/python/kernel_tests/v1_compat_tests:scatter_nd_ops_test_cpu PASSED in 6.9s //tensorflow/python/kernel_tests/v1_compat_tests:session_ops_test_cpu PASSED in 7.9s //tensorflow/python/kernel_tests/v1_compat_tests:stack_op_test_cpu PASSED in 6.4s //tensorflow/python/kernel_tests/variables:dense_update_ops_no_tsan_test_cpu PASSED in 8.3s //tensorflow/python/kernel_tests/variables:dense_update_ops_test_cpu PASSED in 7.4s //tensorflow/python/kernel_tests/variables:partitioned_variables_test PASSED in 10.7s //tensorflow/python/kernel_tests/variables:resource_variable_ops_test_cpu PASSED in 44.3s //tensorflow/python/kernel_tests/variables:variable_ops_test_cpu PASSED in 7.8s //tensorflow/python/kernel_tests/variables:variable_scope_test PASSED in 37.2s //tensorflow/python/kernel_tests/variables:variables_test PASSED in 9.0s //tensorflow/python/lib/core:custom_float_test PASSED in 9.6s //tensorflow/python/lib/io:file_io_test PASSED in 9.0s //tensorflow/python/lib/io:tf_record_test PASSED in 8.6s //tensorflow/python/module:module_test PASSED in 8.4s //tensorflow/python/ops/losses:util_test PASSED in 8.1s //tensorflow/python/ops/memory_tests:custom_gradient_memory_test_cpu PASSED in 10.0s //tensorflow/python/ops/numpy_ops:np_array_ops_test_cpu PASSED in 62.8s //tensorflow/python/ops/numpy_ops:np_arrays_test_cpu PASSED in 6.7s //tensorflow/python/ops/numpy_ops:np_dtypes_test_cpu PASSED in 7.3s //tensorflow/python/ops/numpy_ops:np_interop_test_cpu PASSED in 35.3s //tensorflow/python/ops/numpy_ops:np_logic_test_cpu PASSED in 10.5s //tensorflow/python/ops/numpy_ops:np_math_ops_test_cpu PASSED in 20.4s //tensorflow/python/ops/numpy_ops:np_random_test_cpu PASSED in 62.3s //tensorflow/python/ops/numpy_ops:np_utils_test_cpu PASSED in 6.0s //tensorflow/python/ops/numpy_ops/integration_test:np_config_test_cpu PASSED in 11.5s //tensorflow/python/ops/numpy_ops/integration_test:public_symbol_test PASSED in 11.9s //tensorflow/python/ops/parallel_for:array_test_cpu PASSED in 31.6s //tensorflow/python/ops/parallel_for:gradients_test_cpu PASSED in 8.8s //tensorflow/python/ops/parallel_for:xla_control_flow_ops_test_cpu PASSED in 46.1s //tensorflow/python/ops/ragged:convert_to_tensor_or_ragged_tensor_op_test PASSED in 6.1s //tensorflow/python/ops/ragged:ragged_batch_gather_op_test PASSED in 35.1s //tensorflow/python/ops/ragged:ragged_bitcast_op_test PASSED in 5.7s //tensorflow/python/ops/ragged:ragged_boolean_mask_op_test PASSED in 14.7s //tensorflow/python/ops/ragged:ragged_concat_op_test PASSED in 9.1s //tensorflow/python/ops/ragged:ragged_const_op_test PASSED in 11.4s //tensorflow/python/ops/ragged:ragged_constant_value_op_test PASSED in 6.4s //tensorflow/python/ops/ragged:ragged_cross_op_test PASSED in 18.2s //tensorflow/python/ops/ragged:ragged_dispatch_test PASSED in 114.8s //tensorflow/python/ops/ragged:ragged_dynamic_partition_op_test_cpu PASSED in 14.7s //tensorflow/python/ops/ragged:ragged_eager_test PASSED in 5.9s //tensorflow/python/ops/ragged:ragged_expand_dims_op_test PASSED in 9.1s //tensorflow/python/ops/ragged:ragged_factory_ops_test_cpu PASSED in 13.8s //tensorflow/python/ops/ragged:ragged_from_sparse_op_test PASSED in 7.4s //tensorflow/python/ops/ragged:ragged_from_tensor_op_test PASSED in 22.3s //tensorflow/python/ops/ragged:ragged_gather_nd_op_test PASSED in 10.3s //tensorflow/python/ops/ragged:ragged_map_flat_values_op_test PASSED in 7.8s //tensorflow/python/ops/ragged:ragged_map_fn_op_test PASSED in 12.6s //tensorflow/python/ops/ragged:ragged_math_ops_test PASSED in 11.8s //tensorflow/python/ops/ragged:ragged_matmul_op_test PASSED in 36.5s //tensorflow/python/ops/ragged:ragged_merge_dims_op_test PASSED in 23.2s //tensorflow/python/ops/ragged:ragged_one_hot_op_test PASSED in 8.3s //tensorflow/python/ops/ragged:ragged_operators_test PASSED in 19.0s //tensorflow/python/ops/ragged:ragged_placeholder_op_test PASSED in 6.4s //tensorflow/python/ops/ragged:ragged_print_op_test PASSED in 14.1s //tensorflow/python/ops/ragged:ragged_range_op_test PASSED in 7.4s //tensorflow/python/ops/ragged:ragged_rank_op_test PASSED in 6.3s //tensorflow/python/ops/ragged:ragged_reduce_op_test PASSED in 31.3s //tensorflow/python/ops/ragged:ragged_resize_image_op_test PASSED in 15.7s //tensorflow/python/ops/ragged:ragged_reverse_op_test PASSED in 7.5s //tensorflow/python/ops/ragged:ragged_row_lengths_op_test PASSED in 9.4s //tensorflow/python/ops/ragged:ragged_row_splits_to_segment_ids_op_test PASSED in 6.5s //tensorflow/python/ops/ragged:ragged_segment_ids_to_row_splits_op_test PASSED in 6.8s //tensorflow/python/ops/ragged:ragged_segment_op_test PASSED in 13.1s //tensorflow/python/ops/ragged:ragged_size_op_test PASSED in 15.1s //tensorflow/python/ops/ragged:ragged_split_op_test PASSED in 34.5s //tensorflow/python/ops/ragged:ragged_squeeze_op_test PASSED in 15.4s //tensorflow/python/ops/ragged:ragged_stack_op_test PASSED in 11.8s //tensorflow/python/ops/ragged:ragged_tensor_bounding_shape_op_test PASSED in 11.5s //tensorflow/python/ops/ragged:ragged_tensor_shape_test PASSED in 54.0s //tensorflow/python/ops/ragged:ragged_tile_op_test PASSED in 45.0s //tensorflow/python/ops/ragged:ragged_to_sparse_op_test PASSED in 7.1s //tensorflow/python/ops/ragged:ragged_to_tensor_op_test PASSED in 63.7s //tensorflow/python/ops/ragged:ragged_util_test PASSED in 19.3s //tensorflow/python/ops/ragged:ragged_where_op_test PASSED in 28.9s //tensorflow/python/ops/ragged:row_partition_test PASSED in 24.1s //tensorflow/python/ops/ragged:string_ngrams_op_test PASSED in 6.2s //tensorflow/python/ops/ragged:strings_reduce_join_op_test PASSED in 7.0s //tensorflow/python/ops/structured:structured_array_ops_test PASSED in 40.6s //tensorflow/python/ops/structured:structured_tensor_slice_test PASSED in 55.0s //tensorflow/python/ops/structured:structured_tensor_spec_test PASSED in 9.3s //tensorflow/python/ops/structured:structured_tensor_test PASSED in 36.9s //tensorflow/python/ops/v1_compat_tests:gradient_checker_test_cpu PASSED in 10.1s //tensorflow/python/platform:benchmark_test PASSED in 6.5s //tensorflow/python/platform:build_info_test PASSED in 6.1s //tensorflow/python/platform:resource_loader_test PASSED in 1.9s //tensorflow/python/profiler:pprof_profiler_test PASSED in 5.7s //tensorflow/python/profiler:profile_context_test_cpu PASSED in 29.8s //tensorflow/python/profiler:profiler_client_test_cpu PASSED in 6.7s //tensorflow/python/profiler:profiler_test_cpu PASSED in 15.1s //tensorflow/python/profiler:profiler_v2_test_cpu PASSED in 7.6s //tensorflow/python/profiler:profiler_wrapper_test PASSED in 5.4s //tensorflow/python/profiler:tfprof_logger_test PASSED in 6.4s //tensorflow/python/profiler/integration_test:profiler_api_test_cpu PASSED in 22.8s //tensorflow/python/profiler/internal:flops_registry_test PASSED in 9.2s //tensorflow/python/profiler/internal:print_model_analysis_test PASSED in 10.0s //tensorflow/python/profiler/internal:run_metadata_test_cpu PASSED in 10.6s //tensorflow/python/saved_model:fingerprinting_test PASSED in 8.0s //tensorflow/python/saved_model:keras_injection_test PASSED in 12.7s //tensorflow/python/saved_model:load_v1_in_v2_test PASSED in 14.0s //tensorflow/python/saved_model:loader_test PASSED in 9.2s //tensorflow/python/saved_model:method_name_updater_test PASSED in 6.2s //tensorflow/python/saved_model:metrics_test PASSED in 8.5s //tensorflow/python/saved_model:nested_structure_coder_test PASSED in 5.9s //tensorflow/python/saved_model:pywrap_saved_model_fingerprinting_test PASSED in 5.7s //tensorflow/python/saved_model:pywrap_saved_model_metrics_test PASSED in 6.9s //tensorflow/python/saved_model:revived_types_test PASSED in 8.1s //tensorflow/python/saved_model:save_context_test PASSED in 5.8s //tensorflow/python/saved_model:save_test PASSED in 22.6s //tensorflow/python/saved_model:saved_model_test PASSED in 19.9s //tensorflow/python/saved_model:signature_def_utils_test PASSED in 6.7s //tensorflow/python/saved_model:simple_save_test PASSED in 8.3s //tensorflow/python/saved_model:tracing_utils_test PASSED in 9.7s //tensorflow/python/saved_model:utils_test PASSED in 7.0s //tensorflow/python/saved_model/model_utils:export_output_test PASSED in 9.5s //tensorflow/python/saved_model/model_utils:export_test PASSED in 10.3s //tensorflow/python/saved_model/model_utils:mode_keys_test PASSED in 5.1s //tensorflow/python/saved_model/registration:registration_saving_test PASSED in 13.7s //tensorflow/python/saved_model/registration:registration_test PASSED in 7.7s //tensorflow/python/saved_model/registration:tf_registration_test PASSED in 13.5s //tensorflow/python/summary:plugin_asset_test PASSED in 5.7s //tensorflow/python/summary:summary_iterator_test PASSED in 6.2s //tensorflow/python/summary:summary_test PASSED in 6.6s //tensorflow/python/summary:summary_v2_test PASSED in 7.2s //tensorflow/python/summary/writer:writer_test PASSED in 17.9s //tensorflow/python/tools:aot_compiled_test PASSED in 18.7s //tensorflow/python/tools:freeze_graph_test PASSED in 14.7s //tensorflow/python/tools:optimize_for_inference_test PASSED in 6.1s //tensorflow/python/tools:print_selective_registration_header_test PASSED in 14.4s //tensorflow/python/tools:saved_model_cli_test PASSED in 27.1s //tensorflow/python/tools:saved_model_utils_test PASSED in 6.0s //tensorflow/python/tools:strip_unused_test PASSED in 6.3s //tensorflow/python/tools/api/generator:create_python_api_test PASSED in 13.8s //tensorflow/python/tools/api/generator:output_init_files_test PASSED in 11.7s //tensorflow/python/tools/api/generator:tensorflow_doc_srcs_test PASSED in 10.1s //tensorflow/python/tpu:bfloat16_test PASSED in 12.6s //tensorflow/python/tpu:feature_column_test PASSED in 13.7s //tensorflow/python/tpu:topology_test PASSED in 6.6s //tensorflow/python/tpu:tpu_embedding_for_serving_test PASSED in 8.0s //tensorflow/python/tpu:tpu_embedding_v2_utils_test PASSED in 6.8s //tensorflow/python/tpu:tpu_infeed_test PASSED in 13.9s //tensorflow/python/tpu:tpu_sharding_test PASSED in 6.9s //tensorflow/python/tpu:tpu_test_wrapper_test PASSED in 6.5s //tensorflow/python/tpu/client:client_py_test PASSED in 6.4s //tensorflow/python/trackable:autotrackable_test PASSED in 7.5s //tensorflow/python/trackable:base_delegate_test PASSED in 9.8s //tensorflow/python/trackable:base_test PASSED in 10.1s //tensorflow/python/trackable:data_structures_test PASSED in 9.6s //tensorflow/python/trackable:python_state_test PASSED in 8.1s //tensorflow/python/trackable:resource_test PASSED in 6.4s //tensorflow/python/trackable:trackable_utils_test PASSED in 6.5s //tensorflow/python/training:adadelta_test_cpu PASSED in 14.5s //tensorflow/python/training:adagrad_da_test_cpu PASSED in 9.2s //tensorflow/python/training:adagrad_test_cpu PASSED in 17.1s //tensorflow/python/training:adam_test_cpu PASSED in 13.3s //tensorflow/python/training:basic_loops_test_cpu PASSED in 7.4s //tensorflow/python/training:basic_session_run_hooks_test PASSED in 19.4s //tensorflow/python/training:checkpoint_ops_test PASSED in 5.9s //tensorflow/python/training:coordinator_test_cpu PASSED in 16.4s //tensorflow/python/training:device_setter_test_cpu PASSED in 6.2s //tensorflow/python/training:ftrl_test_cpu PASSED in 14.2s //tensorflow/python/training:gradient_descent_test_cpu PASSED in 8.2s //tensorflow/python/training:input_test PASSED in 21.8s //tensorflow/python/training:momentum_test_cpu PASSED in 9.6s //tensorflow/python/training:monitored_session_test PASSED in 22.1s //tensorflow/python/training:moving_averages_test_cpu PASSED in 11.9s //tensorflow/python/training:optimizer_test_cpu PASSED in 9.5s //tensorflow/python/training:proximal_adagrad_test_cpu PASSED in 11.1s //tensorflow/python/training:proximal_gradient_descent_test_cpu PASSED in 9.6s //tensorflow/python/training:quantize_training_test_cpu PASSED in 7.1s //tensorflow/python/training:queue_runner_test_cpu PASSED in 6.2s //tensorflow/python/training:rmsprop_test_cpu PASSED in 23.9s //tensorflow/python/training:saver_large_partitioned_variable_test PASSED in 12.7s //tensorflow/python/training:saver_test_2gpu PASSED in 32.7s //tensorflow/python/training:saver_test_cpu PASSED in 30.7s //tensorflow/python/training:server_lib_multiple_containers_test PASSED in 6.8s //tensorflow/python/training:server_lib_same_variables_clear_container_test PASSED in 6.8s //tensorflow/python/training:server_lib_same_variables_clear_test PASSED in 6.5s //tensorflow/python/training:server_lib_same_variables_no_clear_test PASSED in 6.9s //tensorflow/python/training:server_lib_sparse_job_test PASSED in 6.2s //tensorflow/python/training:server_lib_test PASSED in 13.8s //tensorflow/python/training:session_manager_test_cpu PASSED in 75.3s //tensorflow/python/training:slot_creator_test_cpu PASSED in 6.6s //tensorflow/python/training:supervisor_test PASSED in 12.6s //tensorflow/python/training:training_ops_mlir_test_cpu PASSED in 7.1s //tensorflow/python/training:training_ops_test_cpu PASSED in 6.7s //tensorflow/python/training:training_util_test PASSED in 7.3s //tensorflow/python/training:warm_starting_util_test PASSED in 20.4s //tensorflow/python/training/experimental:loss_scale_optimizer_test PASSED in 12.9s //tensorflow/python/training/experimental:loss_scale_test PASSED in 24.2s //tensorflow/python/training/experimental:mixed_precision_test_cpu PASSED in 6.1s //tensorflow/python/training/saving:saveable_object_util_test PASSED in 10.2s //tensorflow/python/util:compat_test PASSED in 5.9s //tensorflow/python/util:decorator_utils_test PASSED in 10.6s //tensorflow/python/util:deprecation_test PASSED in 6.2s //tensorflow/python/util:dispatch_test PASSED in 8.1s //tensorflow/python/util:example_parser_configuration_test PASSED in 8.0s //tensorflow/python/util:fast_module_type_test PASSED in 5.8s //tensorflow/python/util:function_parameter_canonicalizer_test PASSED in 8.6s //tensorflow/python/util:function_utils_test PASSED in 6.2s //tensorflow/python/util:keyword_args_test PASSED in 5.8s //tensorflow/python/util:lock_util_test PASSED in 7.9s //tensorflow/python/util:module_wrapper_test PASSED in 6.7s //tensorflow/python/util:nest_test PASSED in 10.9s //tensorflow/python/util:object_identity_test PASSED in 7.0s //tensorflow/python/util:pywrap_xla_ops_test PASSED in 2.3s //tensorflow/python/util:serialization_test PASSED in 5.8s //tensorflow/python/util:tf_contextlib_test PASSED in 6.7s //tensorflow/python/util:tf_decorator_test PASSED in 6.4s //tensorflow/python/util:tf_export_test PASSED in 6.4s //tensorflow/python/util:tf_inspect_test PASSED in 6.6s //tensorflow/python/util:tf_should_use_test PASSED in 9.2s //tensorflow/python/util:tf_stack_test PASSED in 6.6s //tensorflow/python/util:traceback_utils_test PASSED in 11.1s //tensorflow/python/util:type_annotations_test PASSED in 6.2s //tensorflow/python/util:variable_utils_test PASSED in 7.0s //tensorflow/python/util:vlog_test PASSED in 15.2s //tensorflow/tools/api/tests:module_test PASSED in 13.1s //tensorflow/tools/benchmark:benchmark_model_test PASSED in 2.3s //tensorflow/tools/common:public_api_test PASSED in 2.0s //tensorflow/tools/common:traverse_test PASSED in 2.0s //tensorflow/tools/compatibility:all_renames_v2_test PASSED in 6.3s //tensorflow/tools/compatibility:ast_edits_test PASSED in 5.4s //tensorflow/tools/compatibility:test_file_v1_0 PASSED in 11.4s //tensorflow/tools/compatibility:test_file_v2_0 PASSED in 14.2s //tensorflow/tools/compatibility:tf_upgrade_test PASSED in 5.9s //tensorflow/tools/compatibility:tf_upgrade_v2_safety_test PASSED in 5.9s //tensorflow/tools/docs:tf_doctest_test PASSED in 0.9s //tensorflow/tools/graph_transforms:file_utils_test PASSED in 0.5s //tensorflow/tools/graph_transforms:transform_graph_test PASSED in 2.5s //tensorflow/tools/graph_transforms:transform_utils_test PASSED in 2.7s //tensorflow/tools/graph_transforms:transforms_test PASSED in 4.0s //tensorflow/tools/proto_text:gen_proto_text_functions_lib_test PASSED in 0.2s //tensorflow/tools/tensorflow_builder/compat_checker:compat_checker_test PASSED in 0.4s //tensorflow/tsl/c:tsl_status_helper_test PASSED in 0.3s //tensorflow/tsl/c:tsl_status_test PASSED in 0.1s //tensorflow/tsl/concurrency:async_value_ref_test PASSED in 0.1s //tensorflow/tsl/concurrency:async_value_test PASSED in 0.1s //tensorflow/tsl/concurrency:concurrent_vector_test PASSED in 0.2s //tensorflow/tsl/cuda:cudnn_version_test PASSED in 0.1s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_agent_test PASSED in 12.8s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_error_util_test PASSED in 0.1s //tensorflow/tsl/distributed_runtime/coordination:coordination_service_recoverable_job_test PASSED in 0.2s //tensorflow/tsl/distributed_runtime/preemption:preemption_notifier_test PASSED in 7.2s //tensorflow/tsl/distributed_runtime/preemption:preemption_sync_manager_test PASSED in 5.6s //tensorflow/tsl/distributed_runtime/rpc:grpc_channel_test PASSED in 0.3s //tensorflow/tsl/distributed_runtime/rpc:grpc_util_test PASSED in 0.2s //tensorflow/tsl/framework:cancellation_test PASSED in 1.2s //tensorflow/tsl/framework:device_id_utils_test PASSED in 3.8s //tensorflow/tsl/framework/convolution:spatial_convolutions_test PASSED in 0.2s //tensorflow/tsl/lib/gtl:tsl_lib_gtl_tests PASSED in 0.3s //tensorflow/tsl/lib/hash:crc32c_test PASSED in 0.2s //tensorflow/tsl/lib/histogram:histogram_test PASSED in 0.1s //tensorflow/tsl/lib/io:buffered_inputstream_test PASSED in 0.2s //tensorflow/tsl/lib/io:cache_test PASSED in 0.1s //tensorflow/tsl/lib/io:inputbuffer_test PASSED in 1.2s //tensorflow/tsl/lib/io:inputstream_interface_test PASSED in 0.6s //tensorflow/tsl/lib/io:random_inputstream_test PASSED in 0.1s //tensorflow/tsl/lib/io:record_reader_writer_test PASSED in 0.4s //tensorflow/tsl/lib/io:recordio_test PASSED in 0.3s //tensorflow/tsl/lib/io:table_test PASSED in 3.7s //tensorflow/tsl/lib/io:zlib_buffers_test PASSED in 2.0s //tensorflow/tsl/lib/io/snappy:snappy_test PASSED in 0.6s //tensorflow/tsl/lib/math:math_util_test PASSED in 0.1s //tensorflow/tsl/lib/random:distribution_sampler_test PASSED in 0.1s //tensorflow/tsl/lib/random:philox_random_test PASSED in 0.1s //tensorflow/tsl/lib/random:random_distributions_test PASSED in 16.3s //tensorflow/tsl/lib/random:simple_philox_test PASSED in 0.2s //tensorflow/tsl/lib/random:weighted_picker_test PASSED in 13.3s //tensorflow/tsl/platform:ctstring_test PASSED in 0.2s //tensorflow/tsl/platform:denormal_test PASSED in 0.1s //tensorflow/tsl/platform:errors_test PASSED in 0.1s //tensorflow/tsl/platform:fingerprint_test PASSED in 0.7s //tensorflow/tsl/platform:float8_test PASSED in 7.1s //tensorflow/tsl/platform:hash_test PASSED in 0.4s //tensorflow/tsl/platform:integral_types_test PASSED in 0.3s //tensorflow/tsl/platform:intrusive_ptr_test PASSED in 0.1s //tensorflow/tsl/platform:logging_test PASSED in 25.4s //tensorflow/tsl/platform:mutex_test PASSED in 0.4s //tensorflow/tsl/platform:net_test PASSED in 0.1s //tensorflow/tsl/platform:numbers_test PASSED in 0.1s //tensorflow/tsl/platform:path_test PASSED in 0.2s //tensorflow/tsl/platform:port_test PASSED in 8.2s //tensorflow/tsl/platform:random_test PASSED in 2.7s //tensorflow/tsl/platform:refcount_test PASSED in 0.2s //tensorflow/tsl/platform:retrying_file_system_test PASSED in 0.1s //tensorflow/tsl/platform:retrying_utils_test PASSED in 0.2s //tensorflow/tsl/platform:scanner_test PASSED in 0.1s //tensorflow/tsl/platform:setround_test PASSED in 0.2s //tensorflow/tsl/platform:stacktrace_handler_test PASSED in 2.5s //tensorflow/tsl/platform:stacktrace_test PASSED in 0.2s //tensorflow/tsl/platform:status_matchers_test PASSED in 0.2s //tensorflow/tsl/platform:status_test PASSED in 0.1s //tensorflow/tsl/platform:statusor_test PASSED in 17.9s //tensorflow/tsl/platform:str_util_test PASSED in 0.3s //tensorflow/tsl/platform:strcat_test PASSED in 0.2s //tensorflow/tsl/platform:stringpiece_test PASSED in 0.2s //tensorflow/tsl/platform:stringprintf_test PASSED in 0.1s //tensorflow/tsl/platform:subprocess_test PASSED in 0.4s //tensorflow/tsl/platform:tstring_test PASSED in 0.1s //tensorflow/tsl/platform:unbounded_work_queue_test PASSED in 0.6s //tensorflow/tsl/platform/cloud:compute_engine_metadata_client_test PASSED in 0.1s //tensorflow/tsl/platform/cloud:compute_engine_zone_provider_test PASSED in 0.1s //tensorflow/tsl/platform/cloud:curl_http_request_test PASSED in 8.3s //tensorflow/tsl/platform/cloud:expiring_lru_cache_test PASSED in 0.3s //tensorflow/tsl/platform/cloud:gcs_dns_cache_test PASSED in 0.2s //tensorflow/tsl/platform/cloud:gcs_file_system_test PASSED in 4.9s //tensorflow/tsl/platform/cloud:gcs_throttle_test PASSED in 0.3s //tensorflow/tsl/platform/cloud:google_auth_provider_test PASSED in 0.2s //tensorflow/tsl/platform/cloud:oauth_client_test PASSED in 0.2s //tensorflow/tsl/platform/cloud:ram_file_block_cache_test PASSED in 2.5s //tensorflow/tsl/platform/cloud:time_util_test PASSED in 0.1s //tensorflow/tsl/profiler/backends/cpu:traceme_recorder_test PASSED in 0.2s //tensorflow/tsl/profiler/convert:trace_container_test PASSED in 0.1s //tensorflow/tsl/profiler/convert:trace_events_to_json_test PASSED in 0.2s //tensorflow/tsl/profiler/convert:xla_op_utils_test PASSED in 0.1s //tensorflow/tsl/profiler/convert:xplane_to_trace_events_test PASSED in 0.1s //tensorflow/tsl/profiler/lib:profiler_factory_test PASSED in 0.2s //tensorflow/tsl/profiler/lib:profiler_lock_test PASSED in 0.3s //tensorflow/tsl/profiler/lib:scoped_annotation_test PASSED in 0.1s //tensorflow/tsl/profiler/lib:traceme_encode_test PASSED in 0.1s //tensorflow/tsl/profiler/rpc/client:profiler_client_test PASSED in 3.5s //tensorflow/tsl/profiler/rpc/client:remote_profiler_session_manager_test PASSED in 3.3s //tensorflow/tsl/profiler/utils:buffer_pool_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:group_events_test PASSED in 0.3s //tensorflow/tsl/profiler/utils:parse_annotation_test PASSED in 0.6s //tensorflow/tsl/profiler/utils:preprocess_xplane_test PASSED in 0.4s //tensorflow/tsl/profiler/utils:tf_op_utils_test PASSED in 0.2s //tensorflow/tsl/profiler/utils:timespan_test PASSED in 0.1s //tensorflow/tsl/profiler/utils:tpu_xplane_utils_test PASSED in 0.2s //tensorflow/tsl/profiler/utils:xplane_builder_test PASSED in 0.2s //tensorflow/tsl/profiler/utils:xplane_utils_test PASSED in 0.1s //tensorflow/tsl/util:device_name_utils_test PASSED in 0.1s //tensorflow/tsl/util:stats_calculator_test PASSED in 0.1s //tensorflow/compiler/tests:complex_div_test_cpu PASSED in 9.5s Stats over 2 runs: max = 9.5s, min = 8.5s, avg = 9.0s, dev = 0.5s //tensorflow/compiler/tests:complex_div_test_cpu_mlir_bridge_test PASSED in 7.4s Stats over 2 runs: max = 7.4s, min = 6.4s, avg = 6.9s, dev = 0.5s //tensorflow/compiler/xla/tests:conditional_test_cpu PASSED in 8.5s Stats over 2 runs: max = 8.5s, min = 7.9s, avg = 8.2s, dev = 0.3s //tensorflow/python:control_flow_ops_test_cpu PASSED in 26.6s Stats over 2 runs: max = 26.6s, min = 22.8s, avg = 24.7s, dev = 1.9s //tensorflow/python/data/experimental/kernel_tests/optimization:optimization_test PASSED in 18.6s Stats over 2 runs: max = 18.6s, min = 13.5s, avg = 16.1s, dev = 2.5s //tensorflow/python/data/experimental/kernel_tests/service:metadata_test PASSED in 21.9s Stats over 2 runs: max = 21.9s, min = 20.2s, avg = 21.1s, dev = 0.9s //tensorflow/python/data/kernel_tests:padded_batch_test PASSED in 26.5s Stats over 2 runs: max = 26.5s, min = 25.3s, avg = 25.9s, dev = 0.6s //tensorflow/python/data/kernel_tests:repeat_test PASSED in 38.2s Stats over 2 runs: max = 38.2s, min = 36.8s, avg = 37.5s, dev = 0.7s //tensorflow/python/data/kernel_tests:window_test PASSED in 35.7s Stats over 2 runs: max = 35.7s, min = 25.6s, avg = 30.7s, dev = 5.1s //tensorflow/python/distribute:strategy_common_test_2gpu PASSED in 25.2s Stats over 2 runs: max = 25.2s, min = 19.2s, avg = 22.2s, dev = 3.0s //tensorflow/python/distribute:strategy_common_test_cpu PASSED in 22.4s Stats over 2 runs: max = 22.4s, min = 18.9s, avg = 20.7s, dev = 1.7s //tensorflow/python/distribute:strategy_common_test_xla_2gpu PASSED in 12.3s Stats over 2 runs: max = 12.3s, min = 12.2s, avg = 12.2s, dev = 0.1s //tensorflow/python/kernel_tests/array_ops:scatter_nd_ops_test_cpu PASSED in 12.1s Stats over 2 runs: max = 12.1s, min = 11.5s, avg = 11.8s, dev = 0.3s //tensorflow/python/kernel_tests/array_ops:scatter_ops_test_cpu PASSED in 21.8s Stats over 2 runs: max = 21.8s, min = 20.7s, avg = 21.3s, dev = 0.5s //tensorflow/python/kernel_tests/control_flow:functional_ops_test_cpu PASSED in 13.7s Stats over 2 runs: max = 13.7s, min = 13.3s, avg = 13.5s, dev = 0.2s //tensorflow/python/kernel_tests/control_flow:map_fn_test_cpu PASSED in 9.2s Stats over 2 runs: max = 9.2s, min = 7.2s, avg = 8.2s, dev = 1.0s //tensorflow/python/kernel_tests/nn_ops:bias_op_d9m_test_cpu PASSED in 101.7s Stats over 2 runs: max = 101.7s, min = 38.3s, avg = 70.0s, dev = 31.7s //tensorflow/python/kernel_tests/nn_ops:conv2d_backprop_filter_grad_test_cpu PASSED in 137.2s Stats over 2 runs: max = 137.2s, min = 6.4s, avg = 71.8s, dev = 65.4s //tensorflow/compiler/tests:spacetobatch_op_test_cpu PASSED in 8.4s Stats over 3 runs: max = 8.4s, min = 8.1s, avg = 8.3s, dev = 0.1s //tensorflow/compiler/tests:spacetobatch_op_test_cpu_mlir_bridge_test PASSED in 7.9s Stats over 3 runs: max = 7.9s, min = 7.6s, avg = 7.7s, dev = 0.1s //tensorflow/compiler/xla/tests:triangular_solve_test_cpu PASSED in 53.4s Stats over 3 runs: max = 53.4s, min = 51.3s, avg = 52.6s, dev = 0.9s //tensorflow/core/data/service:thread_safe_buffer_test PASSED in 0.3s Stats over 3 runs: max = 0.3s, min = 0.2s, avg = 0.3s, dev = 0.0s //tensorflow/python/data/experimental/kernel_tests/service:multi_process_cluster_test PASSED in 17.4s Stats over 3 runs: max = 17.4s, min = 11.7s, avg = 15.4s, dev = 2.6s //tensorflow/python/data/kernel_tests:unique_test PASSED in 46.2s Stats over 3 runs: max = 46.2s, min = 44.4s, avg = 45.4s, dev = 0.8s //tensorflow/python/kernel_tests/array_ops:gather_op_test_cpu PASSED in 44.7s Stats over 3 runs: max = 44.7s, min = 24.7s, avg = 31.7s, dev = 9.2s //tensorflow/python/kernel_tests/array_ops:weights_broadcast_test PASSED in 8.0s Stats over 3 runs: max = 8.0s, min = 7.9s, avg = 8.0s, dev = 0.1s //tensorflow/python/kernel_tests/distributions:util_test_cpu PASSED in 10.9s Stats over 3 runs: max = 10.9s, min = 9.5s, avg = 10.0s, dev = 0.6s //tensorflow/python/kernel_tests/linalg:matrix_triangular_solve_op_test_cpu PASSED in 114.1s Stats over 3 runs: max = 114.1s, min = 8.9s, avg = 44.4s, dev = 49.3s //tensorflow/python/kernel_tests/random:multinomial_op_big_test_cpu PASSED in 10.6s Stats over 3 runs: max = 10.6s, min = 8.7s, avg = 9.5s, dev = 0.8s //tensorflow/c/eager:c_api_test_cpu FAILED in 3 out of 3 in 34.8s Stats over 3 runs: max = 34.8s, min = 31.7s, avg = 33.6s, dev = 1.4s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/c/eager/c_api_test_cpu/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/c/eager/c_api_test_cpu/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/c/eager/c_api_test_cpu/test_attempts/attempt_2.log //tensorflow/compiler/tests:ternary_ops_test_cpu PASSED in 13.0s Stats over 4 runs: max = 13.0s, min = 9.1s, avg = 10.8s, dev = 1.5s //tensorflow/compiler/tests:ternary_ops_test_cpu_mlir_bridge_test PASSED in 45.7s Stats over 4 runs: max = 45.7s, min = 41.1s, avg = 42.7s, dev = 1.8s //tensorflow/compiler/tests:unary_ops_test_cpu PASSED in 31.7s Stats over 4 runs: max = 31.7s, min = 7.6s, avg = 21.0s, dev = 10.3s //tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test PASSED in 36.4s Stats over 4 runs: max = 36.4s, min = 5.3s, avg = 23.5s, dev = 13.1s //tensorflow/compiler/xla/tests:dynamic_ops_test_cpu PASSED in 8.7s Stats over 4 runs: max = 8.7s, min = 8.1s, avg = 8.4s, dev = 0.2s //tensorflow/core/kernels:example_parsing_ops_test PASSED in 0.5s Stats over 4 runs: max = 0.5s, min = 0.4s, avg = 0.4s, dev = 0.0s //tensorflow/python:nn_batchnorm_test_cpu PASSED in 17.7s Stats over 4 runs: max = 17.7s, min = 12.7s, avg = 15.0s, dev = 1.8s //tensorflow/python:nn_fused_batchnorm_d9m_test_cpu PASSED in 15.6s Stats over 4 runs: max = 15.6s, min = 14.8s, avg = 15.2s, dev = 0.3s //tensorflow/python/data/experimental/kernel_tests:auto_shard_dataset_test PASSED in 111.2s Stats over 4 runs: max = 111.2s, min = 96.8s, avg = 104.1s, dev = 5.6s //tensorflow/python/data/experimental/kernel_tests:map_and_batch_test PASSED in 32.8s Stats over 4 runs: max = 32.8s, min = 20.7s, avg = 24.3s, dev = 5.0s //tensorflow/python/data/experimental/kernel_tests:parse_example_dataset_test PASSED in 26.8s Stats over 4 runs: max = 26.8s, min = 14.6s, avg = 20.4s, dev = 5.7s //tensorflow/python/data/experimental/kernel_tests:rebatch_dataset_test PASSED in 19.9s Stats over 4 runs: max = 19.9s, min = 9.1s, avg = 13.8s, dev = 4.2s //tensorflow/python/data/experimental/kernel_tests:sql_dataset_test PASSED in 21.8s Stats over 4 runs: max = 21.8s, min = 11.7s, avg = 16.2s, dev = 4.6s //tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_ft_test PASSED in 6.5s Stats over 4 runs: max = 6.5s, min = 5.2s, avg = 5.9s, dev = 0.5s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_test PASSED in 39.7s Stats over 4 runs: max = 39.7s, min = 23.4s, avg = 31.4s, dev = 7.5s //tensorflow/python/data/kernel_tests:batch_test PASSED in 37.4s Stats over 4 runs: max = 37.4s, min = 32.7s, avg = 34.6s, dev = 1.8s //tensorflow/python/data/kernel_tests:fixed_length_record_dataset_test PASSED in 15.3s Stats over 4 runs: max = 15.3s, min = 8.5s, avg = 12.0s, dev = 3.2s //tensorflow/python/data/kernel_tests:from_generator_test PASSED in 22.9s Stats over 4 runs: max = 22.9s, min = 15.7s, avg = 19.0s, dev = 3.0s //tensorflow/python/data/kernel_tests:group_by_window_test PASSED in 18.7s Stats over 4 runs: max = 18.7s, min = 5.7s, avg = 10.9s, dev = 5.4s //tensorflow/python/data/kernel_tests:ragged_batch_test PASSED in 18.1s Stats over 4 runs: max = 18.1s, min = 16.5s, avg = 17.2s, dev = 0.6s //tensorflow/python/data/kernel_tests:skip_test PASSED in 22.1s Stats over 4 runs: max = 22.1s, min = 14.2s, avg = 18.2s, dev = 3.3s //tensorflow/python/data/kernel_tests:take_test PASSED in 21.6s Stats over 4 runs: max = 21.6s, min = 19.5s, avg = 20.3s, dev = 0.8s //tensorflow/python/data/kernel_tests:take_while_test PASSED in 24.4s Stats over 4 runs: max = 24.4s, min = 22.5s, avg = 23.3s, dev = 0.7s //tensorflow/python/data/kernel_tests:text_line_dataset_test PASSED in 27.2s Stats over 4 runs: max = 27.2s, min = 18.7s, avg = 23.1s, dev = 3.9s //tensorflow/python/data/kernel_tests:zip_test PASSED in 15.0s Stats over 4 runs: max = 15.0s, min = 10.6s, avg = 12.2s, dev = 1.7s //tensorflow/python/debug/lib:dumping_callback_test_cpu PASSED in 13.9s Stats over 4 runs: max = 13.9s, min = 13.2s, avg = 13.5s, dev = 0.3s //tensorflow/python/distribute:cross_device_ops_test_2gpu PASSED in 27.9s Stats over 4 runs: max = 27.9s, min = 18.8s, avg = 21.9s, dev = 3.6s //tensorflow/python/distribute:cross_device_ops_test_cpu PASSED in 25.9s Stats over 4 runs: max = 25.9s, min = 18.1s, avg = 21.7s, dev = 3.0s //tensorflow/python/distribute:strategy_gather_test_2gpu PASSED in 20.2s Stats over 4 runs: max = 20.2s, min = 12.8s, avg = 16.8s, dev = 3.3s //tensorflow/python/distribute:strategy_gather_test_cpu PASSED in 25.2s Stats over 4 runs: max = 25.2s, min = 17.4s, avg = 21.4s, dev = 3.7s //tensorflow/python/distribute:strategy_gather_test_xla_2gpu PASSED in 17.8s Stats over 4 runs: max = 17.8s, min = 7.8s, avg = 12.6s, dev = 4.6s //tensorflow/python/framework:convert_to_constants_test PASSED in 102.6s Stats over 4 runs: max = 102.6s, min = 95.9s, avg = 97.9s, dev = 2.7s //tensorflow/python/kernel_tests:collective_ops_test_2gpu PASSED in 38.2s Stats over 4 runs: max = 38.2s, min = 34.5s, avg = 36.5s, dev = 1.5s //tensorflow/python/kernel_tests:collective_ops_test_cpu PASSED in 30.2s Stats over 4 runs: max = 30.2s, min = 25.7s, avg = 28.0s, dev = 1.6s //tensorflow/python/kernel_tests/array_ops:concat_op_test_cpu PASSED in 107.2s Stats over 4 runs: max = 107.2s, min = 104.4s, avg = 105.5s, dev = 1.0s //tensorflow/python/kernel_tests/array_ops:init_ops_test_cpu PASSED in 80.3s Stats over 4 runs: max = 80.3s, min = 32.6s, avg = 55.1s, dev = 20.6s //tensorflow/python/kernel_tests/array_ops:split_op_test_cpu PASSED in 25.6s Stats over 4 runs: max = 25.6s, min = 7.8s, avg = 14.3s, dev = 7.3s //tensorflow/python/kernel_tests/linalg:einsum_op_test_cpu PASSED in 107.3s Stats over 4 runs: max = 107.3s, min = 20.9s, avg = 55.3s, dev = 35.7s //tensorflow/python/kernel_tests/linalg:linear_operator_lower_triangular_test_cpu PASSED in 24.9s Stats over 4 runs: max = 24.9s, min = 23.4s, avg = 24.2s, dev = 0.7s //tensorflow/python/kernel_tests/random:random_gamma_test_cpu PASSED in 73.5s Stats over 4 runs: max = 73.5s, min = 6.8s, avg = 34.7s, dev = 28.8s //tensorflow/python/kernel_tests/signal:window_ops_test_cpu PASSED in 19.1s Stats over 4 runs: max = 19.1s, min = 15.0s, avg = 17.7s, dev = 1.6s //tensorflow/python/ops/ragged:ragged_gather_op_test PASSED in 64.9s Stats over 4 runs: max = 64.9s, min = 14.5s, avg = 36.9s, dev = 18.1s //tensorflow/python/ops/ragged:ragged_getitem_test PASSED in 43.0s Stats over 4 runs: max = 43.0s, min = 38.6s, avg = 41.2s, dev = 1.6s //tensorflow/compiler/tests:async_comp_test_cpu PASSED in 21.0s Stats over 5 runs: max = 21.0s, min = 18.1s, avg = 19.3s, dev = 1.4s //tensorflow/compiler/tests:conv3d_test_cpu PASSED in 11.7s Stats over 5 runs: max = 11.7s, min = 5.8s, avg = 8.3s, dev = 2.7s //tensorflow/compiler/tests:conv3d_test_cpu_mlir_bridge_test PASSED in 12.5s Stats over 5 runs: max = 12.5s, min = 5.6s, avg = 8.6s, dev = 3.2s //tensorflow/compiler/tests:depthwise_conv_op_test_cpu PASSED in 14.0s Stats over 5 runs: max = 14.0s, min = 7.8s, avg = 10.6s, dev = 2.4s //tensorflow/compiler/tests:depthwise_conv_op_test_cpu_mlir_bridge_test PASSED in 14.0s Stats over 5 runs: max = 14.0s, min = 8.4s, avg = 10.8s, dev = 2.3s //tensorflow/compiler/tests:fused_batchnorm_test_cpu PASSED in 7.8s Stats over 5 runs: max = 7.8s, min = 6.6s, avg = 7.1s, dev = 0.4s //tensorflow/compiler/tests:fused_batchnorm_test_cpu_mlir_bridge_test PASSED in 7.9s Stats over 5 runs: max = 7.9s, min = 6.9s, avg = 7.4s, dev = 0.4s //tensorflow/compiler/tests:image_ops_jit_compile_test_cpu PASSED in 8.0s Stats over 5 runs: max = 8.0s, min = 6.8s, avg = 7.2s, dev = 0.4s //tensorflow/compiler/tests:reduce_ops_test_cpu PASSED in 9.9s Stats over 5 runs: max = 9.9s, min = 8.8s, avg = 9.2s, dev = 0.4s //tensorflow/compiler/tests:reduce_ops_test_cpu_mlir_bridge_test PASSED in 13.7s Stats over 5 runs: max = 13.7s, min = 11.4s, avg = 12.7s, dev = 0.8s //tensorflow/compiler/tests:repeat_op_test_cpu PASSED in 7.2s Stats over 5 runs: max = 7.2s, min = 6.3s, avg = 6.5s, dev = 0.3s //tensorflow/compiler/tests:repeat_op_test_cpu_mlir_bridge_test PASSED in 7.1s Stats over 5 runs: max = 7.1s, min = 5.4s, avg = 6.2s, dev = 0.6s //tensorflow/compiler/tests:special_math_test_cpu PASSED in 102.8s Stats over 5 runs: max = 102.8s, min = 12.2s, avg = 44.9s, dev = 31.2s //tensorflow/compiler/tests:special_math_test_cpu_mlir_bridge_test PASSED in 91.3s Stats over 5 runs: max = 91.3s, min = 10.3s, avg = 41.7s, dev = 27.6s //tensorflow/compiler/xla/client/lib:self_adjoint_eig_test_cpu PASSED in 22.6s Stats over 5 runs: max = 22.6s, min = 10.7s, avg = 17.5s, dev = 5.2s //tensorflow/core/grappler/optimizers:constant_folding_test PASSED in 2.2s Stats over 5 runs: max = 2.2s, min = 1.7s, avg = 1.9s, dev = 0.2s //tensorflow/dtensor/python/tests:layout_propagation_test_cpu PASSED in 10.4s Stats over 5 runs: max = 10.4s, min = 8.3s, avg = 9.0s, dev = 0.8s //tensorflow/python/distribute:mirrored_strategy_test_2gpu PASSED in 11.0s Stats over 5 runs: max = 11.0s, min = 9.4s, avg = 10.3s, dev = 0.5s //tensorflow/python/distribute:mirrored_strategy_test_cpu PASSED in 11.1s Stats over 5 runs: max = 11.1s, min = 9.7s, avg = 10.3s, dev = 0.5s //tensorflow/python/distribute:moving_averages_test_2gpu PASSED in 15.8s Stats over 5 runs: max = 15.8s, min = 13.2s, avg = 14.3s, dev = 0.8s //tensorflow/python/distribute:moving_averages_test_cpu PASSED in 14.0s Stats over 5 runs: max = 14.0s, min = 10.1s, avg = 12.5s, dev = 1.3s //tensorflow/python/distribute:vars_test_2gpu PASSED in 14.4s Stats over 5 runs: max = 14.4s, min = 11.8s, avg = 13.1s, dev = 0.9s //tensorflow/python/distribute:vars_test_cpu PASSED in 13.0s Stats over 5 runs: max = 13.0s, min = 11.8s, avg = 12.4s, dev = 0.4s //tensorflow/python/eager:device_placement_test_cpu PASSED in 7.8s Stats over 5 runs: max = 7.8s, min = 6.3s, avg = 7.0s, dev = 0.5s //tensorflow/python/eager:forwardprop_test_cpu PASSED in 124.7s Stats over 5 runs: max = 124.7s, min = 15.0s, avg = 48.0s, dev = 39.2s //tensorflow/python/eager/polymorphic_function:gradients_test_cpu PASSED in 15.6s Stats over 5 runs: max = 15.6s, min = 10.1s, avg = 12.2s, dev = 2.0s //tensorflow/python/kernel_tests/linalg:cholesky_op_test_cpu PASSED in 78.3s Stats over 5 runs: max = 78.3s, min = 33.9s, avg = 48.3s, dev = 16.0s //tensorflow/python/kernel_tests/linalg:linear_operator_adjoint_test_cpu PASSED in 22.6s Stats over 5 runs: max = 22.6s, min = 21.6s, avg = 22.1s, dev = 0.3s //tensorflow/python/kernel_tests/linalg:linear_operator_composition_test_cpu PASSED in 38.1s Stats over 5 runs: max = 38.1s, min = 33.7s, avg = 35.9s, dev = 1.4s //tensorflow/python/kernel_tests/linalg:linear_operator_diag_test_cpu PASSED in 20.7s Stats over 5 runs: max = 20.7s, min = 17.8s, avg = 19.2s, dev = 1.0s //tensorflow/python/kernel_tests/linalg:linear_operator_full_matrix_test_cpu PASSED in 24.7s Stats over 5 runs: max = 24.7s, min = 24.2s, avg = 24.3s, dev = 0.2s //tensorflow/python/kernel_tests/linalg:linear_operator_householder_test_cpu PASSED in 22.5s Stats over 5 runs: max = 22.5s, min = 20.8s, avg = 21.8s, dev = 0.5s //tensorflow/python/kernel_tests/linalg:linear_operator_identity_test_cpu PASSED in 36.5s Stats over 5 runs: max = 36.5s, min = 32.7s, avg = 34.3s, dev = 1.5s //tensorflow/python/kernel_tests/linalg:linear_operator_inversion_test_cpu PASSED in 21.8s Stats over 5 runs: max = 21.8s, min = 20.9s, avg = 21.3s, dev = 0.3s //tensorflow/python/kernel_tests/linalg:linear_operator_permutation_test_cpu PASSED in 19.5s Stats over 5 runs: max = 19.5s, min = 18.5s, avg = 19.0s, dev = 0.3s //tensorflow/python/kernel_tests/linalg:linear_operator_toeplitz_test_cpu PASSED in 16.5s Stats over 5 runs: max = 16.5s, min = 8.3s, avg = 14.2s, dev = 3.0s //tensorflow/python/kernel_tests/linalg:linear_operator_tridiag_test_cpu PASSED in 70.1s Stats over 5 runs: max = 70.1s, min = 68.5s, avg = 69.1s, dev = 0.5s //tensorflow/python/kernel_tests/linalg:linear_operator_util_test_cpu PASSED in 6.6s Stats over 5 runs: max = 6.6s, min = 3.2s, avg = 4.3s, dev = 1.3s //tensorflow/python/kernel_tests/linalg:linear_operator_zeros_test_cpu PASSED in 12.6s Stats over 5 runs: max = 12.6s, min = 10.1s, avg = 11.1s, dev = 0.9s //tensorflow/python/kernel_tests/nn_ops:fractional_avg_pool_op_test PASSED in 13.4s Stats over 5 runs: max = 13.4s, min = 6.2s, avg = 8.1s, dev = 2.7s //tensorflow/python/kernel_tests/nn_ops:fractional_max_pool_op_test PASSED in 14.6s Stats over 5 runs: max = 14.6s, min = 6.2s, avg = 8.5s, dev = 3.2s //tensorflow/python/kernel_tests/sparse_ops:sparse_ops_test_cpu PASSED in 27.5s Stats over 5 runs: max = 27.5s, min = 6.3s, avg = 11.1s, dev = 8.2s //tensorflow/python/ops/parallel_for:math_test_cpu PASSED in 71.6s Stats over 5 runs: max = 71.6s, min = 23.7s, avg = 43.5s, dev = 16.6s //tensorflow/compiler/tests:scan_ops_test_cpu PASSED in 11.1s Stats over 6 runs: max = 11.1s, min = 7.9s, avg = 9.7s, dev = 1.2s //tensorflow/compiler/tests:scan_ops_test_cpu_mlir_bridge_test PASSED in 16.5s Stats over 6 runs: max = 16.5s, min = 11.0s, avg = 13.4s, dev = 1.8s //tensorflow/python:accumulate_n_benchmark_cpu PASSED in 7.1s Stats over 6 runs: max = 7.1s, min = 6.4s, avg = 6.8s, dev = 0.2s //tensorflow/python/data/experimental/kernel_tests:make_batched_features_dataset_test PASSED in 36.8s Stats over 6 runs: max = 36.8s, min = 3.1s, avg = 18.6s, dev = 14.5s //tensorflow/python/kernel_tests/array_ops:diag_op_test_cpu PASSED in 68.3s Stats over 6 runs: max = 68.3s, min = 7.8s, avg = 20.1s, dev = 21.6s //tensorflow/python/kernel_tests/math_ops:reduction_ops_test_cpu PASSED in 40.5s Stats over 6 runs: max = 40.5s, min = 18.7s, avg = 28.3s, dev = 7.2s //tensorflow/python/distribute/experimental/rpc:rpc_ops_test PASSED in 15.2s Stats over 7 runs: max = 15.2s, min = 10.6s, avg = 12.3s, dev = 1.7s //tensorflow/compiler/tests:matrix_diag_ops_test_cpu PASSED in 52.7s Stats over 8 runs: max = 52.7s, min = 2.6s, avg = 19.8s, dev = 16.7s //tensorflow/compiler/tests:matrix_diag_ops_test_cpu_mlir_bridge_test PASSED in 62.7s Stats over 8 runs: max = 62.7s, min = 5.5s, avg = 24.6s, dev = 19.7s //tensorflow/python/data/experimental/kernel_tests:csv_dataset_test PASSED in 26.1s Stats over 8 runs: max = 26.1s, min = 4.5s, avg = 13.3s, dev = 6.8s //tensorflow/python/data/experimental/kernel_tests:parallel_interleave_test PASSED in 28.2s Stats over 8 runs: max = 28.2s, min = 9.5s, avg = 18.9s, dev = 6.7s //tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_ft_test PASSED in 53.3s Stats over 8 runs: max = 53.3s, min = 13.7s, avg = 29.8s, dev = 14.8s //tensorflow/python/data/experimental/kernel_tests/service:coordinated_read_test PASSED in 42.8s Stats over 8 runs: max = 42.8s, min = 9.6s, avg = 20.1s, dev = 12.0s //tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_test PASSED in 19.3s Stats over 8 runs: max = 19.3s, min = 5.2s, avg = 10.3s, dev = 4.8s //tensorflow/python/data/experimental/kernel_tests/service:fault_tolerance_test PASSED in 22.3s Stats over 8 runs: max = 22.3s, min = 15.0s, avg = 17.0s, dev = 2.2s //tensorflow/python/data/kernel_tests:filter_test PASSED in 13.2s Stats over 8 runs: max = 13.2s, min = 9.8s, avg = 12.1s, dev = 1.0s //tensorflow/python/data/kernel_tests:flat_map_test PASSED in 24.2s Stats over 8 runs: max = 24.2s, min = 12.4s, avg = 17.1s, dev = 3.6s //tensorflow/python/data/kernel_tests:shard_test PASSED in 20.5s Stats over 8 runs: max = 20.5s, min = 12.4s, avg = 16.9s, dev = 2.6s //tensorflow/python/data/kernel_tests:shuffle_test PASSED in 54.8s Stats over 8 runs: max = 54.8s, min = 25.9s, avg = 31.0s, dev = 9.1s //tensorflow/python/data/kernel_tests:tf_record_dataset_test PASSED in 23.5s Stats over 8 runs: max = 23.5s, min = 14.3s, avg = 18.3s, dev = 2.5s //tensorflow/python/distribute/failure_handling:failure_handler_test PASSED in 56.2s Stats over 8 runs: max = 56.2s, min = 21.5s, avg = 39.0s, dev = 10.8s //tensorflow/python/kernel_tests/linalg:linalg_ops_test_cpu PASSED in 45.3s Stats over 8 runs: max = 45.3s, min = 26.7s, avg = 36.9s, dev = 6.4s //tensorflow/python/kernel_tests/linalg:linear_operator_block_diag_test_cpu PASSED in 53.6s Stats over 8 runs: max = 53.6s, min = 42.7s, avg = 48.9s, dev = 4.5s //tensorflow/python/kernel_tests/linalg:linear_operator_block_lower_triangular_test_cpu PASSED in 117.2s Stats over 8 runs: max = 117.2s, min = 96.5s, avg = 106.4s, dev = 7.1s //tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_d9m_test_cpu PASSED in 56.1s Stats over 8 runs: max = 56.1s, min = 3.5s, avg = 13.9s, dev = 17.2s //tensorflow/python/kernel_tests/nn_ops:depthwise_conv_op_test_cpu PASSED in 6.8s Stats over 8 runs: max = 6.8s, min = 5.0s, avg = 5.8s, dev = 0.5s //tensorflow/python/kernel_tests/signal:fft_ops_test_cpu PASSED in 21.3s Stats over 8 runs: max = 21.3s, min = 7.9s, avg = 13.0s, dev = 5.1s //tensorflow/python/ops/ragged:dynamic_ragged_shape_test PASSED in 54.3s Stats over 8 runs: max = 54.3s, min = 33.8s, avg = 43.1s, dev = 6.1s //tensorflow/python/ops/ragged:ragged_tensor_test PASSED in 21.0s Stats over 8 runs: max = 21.0s, min = 10.1s, avg = 13.8s, dev = 3.2s //tensorflow/dtensor/python/tests:input_util_test FLAKY, failed in 1 out of 9 in 21.5s Stats over 9 runs: max = 21.5s, min = 8.8s, avg = 15.6s, dev = 4.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/dtensor/python/tests/input_util_test/shard_1_of_8/test_attempts/attempt_1.log //tensorflow/python/distribute/failure_handling:gce_failure_handler_test FLAKY, failed in 1 out of 9 in 98.5s Stats over 9 runs: max = 98.5s, min = 15.2s, avg = 39.9s, dev = 29.5s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/distribute/failure_handling/gce_failure_handler_test/shard_3_of_8/test_attempts/attempt_1.log //tensorflow/compiler/tests:bincount_op_test_cpu PASSED in 6.0s Stats over 10 runs: max = 6.0s, min = 2.5s, avg = 4.0s, dev = 1.1s //tensorflow/compiler/tests:conv2d_test_cpu PASSED in 6.5s Stats over 10 runs: max = 6.5s, min = 3.5s, avg = 4.3s, dev = 1.1s //tensorflow/compiler/tests:conv2d_test_cpu_mlir_bridge_test PASSED in 7.7s Stats over 10 runs: max = 7.7s, min = 5.0s, avg = 6.2s, dev = 0.9s //tensorflow/compiler/tests:image_ops_test_cpu PASSED in 14.9s Stats over 10 runs: max = 14.9s, min = 9.0s, avg = 12.4s, dev = 1.8s //tensorflow/compiler/tests:random_ops_test_cpu PASSED in 18.6s Stats over 10 runs: max = 18.6s, min = 12.6s, avg = 15.4s, dev = 1.7s //tensorflow/compiler/tests:random_ops_test_cpu_mlir_bridge_test PASSED in 18.0s Stats over 10 runs: max = 18.0s, min = 9.5s, avg = 14.2s, dev = 2.5s //tensorflow/compiler/tests:stateless_random_ops_test_cpu PASSED in 58.0s Stats over 10 runs: max = 58.0s, min = 31.3s, avg = 45.4s, dev = 8.8s //tensorflow/compiler/tests:stateless_random_ops_test_cpu_mlir_bridge_test PASSED in 62.5s Stats over 10 runs: max = 62.5s, min = 32.9s, avg = 48.3s, dev = 10.0s //tensorflow/compiler/xla/client/lib:svd_test_cpu PASSED in 71.9s Stats over 10 runs: max = 71.9s, min = 7.0s, avg = 25.8s, dev = 24.4s //tensorflow/compiler/xla/client/lib:tridiagonal_test_cpu PASSED in 7.5s Stats over 10 runs: max = 7.5s, min = 6.3s, avg = 7.0s, dev = 0.3s //tensorflow/compiler/xla/service/cpu:cpu_runtime_test PASSED in 12.1s Stats over 10 runs: max = 12.1s, min = 0.9s, avg = 8.6s, dev = 3.9s //tensorflow/python:special_math_ops_test_cpu PASSED in 43.1s Stats over 10 runs: max = 43.1s, min = 5.7s, avg = 12.2s, dev = 10.6s //tensorflow/python/data/kernel_tests:rejection_resample_test PASSED in 20.9s Stats over 10 runs: max = 20.9s, min = 3.8s, avg = 10.3s, dev = 5.8s //tensorflow/python/distribute:input_lib_test_2gpu PASSED in 29.1s Stats over 10 runs: max = 29.1s, min = 18.5s, avg = 22.5s, dev = 3.1s //tensorflow/python/distribute:input_lib_test_cpu PASSED in 26.3s Stats over 10 runs: max = 26.3s, min = 17.6s, avg = 22.0s, dev = 2.8s //tensorflow/python/distribute:input_lib_type_spec_test_2gpu PASSED in 15.6s Stats over 10 runs: max = 15.6s, min = 4.9s, avg = 10.3s, dev = 3.7s //tensorflow/python/distribute:input_lib_type_spec_test_cpu PASSED in 14.8s Stats over 10 runs: max = 14.8s, min = 5.1s, avg = 9.9s, dev = 3.4s //tensorflow/python/framework:config_vgpu_test_2gpu PASSED in 7.0s Stats over 10 runs: max = 7.0s, min = 5.6s, avg = 6.4s, dev = 0.5s //tensorflow/python/framework:config_vgpu_test_cpu PASSED in 6.2s Stats over 10 runs: max = 6.2s, min = 3.9s, avg = 5.1s, dev = 0.8s //tensorflow/python/framework:function_test_cpu PASSED in 54.8s Stats over 10 runs: max = 54.8s, min = 5.1s, avg = 11.9s, dev = 14.5s //tensorflow/python/grappler:cluster_test_cpu PASSED in 6.6s Stats over 10 runs: max = 6.6s, min = 4.5s, avg = 5.6s, dev = 0.7s //tensorflow/python/kernel_tests/array_ops:array_ops_test_cpu PASSED in 11.7s Stats over 10 runs: max = 11.7s, min = 7.1s, avg = 8.9s, dev = 1.4s //tensorflow/python/kernel_tests/array_ops:inplace_ops_test_cpu PASSED in 7.0s Stats over 10 runs: max = 7.0s, min = 5.9s, avg = 6.5s, dev = 0.4s //tensorflow/python/kernel_tests/data_structures:tensor_array_ops_test_cpu PASSED in 9.8s Stats over 10 runs: max = 9.8s, min = 6.1s, avg = 7.5s, dev = 1.1s //tensorflow/python/kernel_tests/linalg:linear_operator_kronecker_test_cpu PASSED in 27.7s Stats over 10 runs: max = 27.7s, min = 21.7s, avg = 24.3s, dev = 2.0s //tensorflow/python/kernel_tests/linalg:linear_operator_low_rank_update_test_cpu PASSED in 67.0s Stats over 10 runs: max = 67.0s, min = 62.6s, avg = 64.5s, dev = 1.4s //tensorflow/python/kernel_tests/linalg:tridiagonal_matmul_op_test_cpu PASSED in 118.1s Stats over 10 runs: max = 118.1s, min = 5.2s, avg = 17.2s, dev = 33.6s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_ops_test_cpu PASSED in 38.8s Stats over 10 runs: max = 38.8s, min = 7.0s, avg = 22.6s, dev = 10.8s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_test_cpu PASSED in 30.5s Stats over 10 runs: max = 30.5s, min = 7.0s, avg = 17.2s, dev = 8.2s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_test_cpu PASSED in 30.9s Stats over 10 runs: max = 30.9s, min = 6.9s, avg = 12.7s, dev = 9.1s //tensorflow/python/kernel_tests/nn_ops:rnn_test_cpu PASSED in 9.4s Stats over 10 runs: max = 9.4s, min = 4.3s, avg = 7.2s, dev = 1.6s //tensorflow/python/kernel_tests/random:random_index_shuffle_test PASSED in 9.2s Stats over 10 runs: max = 9.2s, min = 8.3s, avg = 8.5s, dev = 0.3s //tensorflow/python/kernel_tests/random:stateless_random_ops_test_cpu PASSED in 80.8s Stats over 10 runs: max = 80.8s, min = 13.6s, avg = 47.1s, dev = 31.3s //tensorflow/python/ops/ragged:ragged_tensor_supported_values_test PASSED in 14.7s Stats over 10 runs: max = 14.7s, min = 13.0s, avg = 14.1s, dev = 0.5s //tensorflow/python/saved_model:load_test_cpu PASSED in 44.3s Stats over 10 runs: max = 44.3s, min = 22.2s, avg = 28.4s, dev = 5.7s //tensorflow/compiler/tests:fft_test_cpu PASSED in 25.3s Stats over 12 runs: max = 25.3s, min = 8.4s, avg = 15.7s, dev = 6.3s //tensorflow/compiler/xla/service:triangular_solve_expander_test PASSED in 5.4s Stats over 12 runs: max = 5.4s, min = 2.6s, avg = 3.9s, dev = 0.8s //tensorflow/python/data/experimental/kernel_tests:group_by_reducer_test PASSED in 36.8s Stats over 12 runs: max = 36.8s, min = 17.0s, avg = 27.9s, dev = 5.1s //tensorflow/python/data/kernel_tests:choose_from_datasets_test PASSED in 10.2s Stats over 12 runs: max = 10.2s, min = 5.0s, avg = 7.5s, dev = 1.7s //tensorflow/python/data/kernel_tests:memory_cleanup_test_cpu PASSED in 16.9s Stats over 12 runs: max = 16.9s, min = 4.4s, avg = 10.4s, dev = 3.3s //tensorflow/python/distribute:multi_process_runner_test_2gpu PASSED in 221.1s Stats over 12 runs: max = 221.1s, min = 10.8s, avg = 48.2s, dev = 58.1s //tensorflow/python/distribute:multi_process_runner_test_cpu PASSED in 219.4s Stats over 12 runs: max = 219.4s, min = 9.6s, avg = 46.9s, dev = 57.9s //tensorflow/python/eager/polymorphic_function:polymorphic_function_test_cpu PASSED in 15.1s Stats over 15 runs: max = 15.1s, min = 8.5s, avg = 12.4s, dev = 1.5s //tensorflow/python/kernel_tests/linalg:linear_operator_circulant_test_cpu PASSED in 53.3s Stats over 15 runs: max = 53.3s, min = 47.8s, avg = 50.1s, dev = 1.7s //tensorflow/python/kernel_tests/nn_ops:rnn_cell_test_cpu PASSED in 44.8s Stats over 15 runs: max = 44.8s, min = 8.0s, avg = 13.3s, dev = 9.2s //tensorflow/python:image_ops_test_cpu PASSED in 14.5s Stats over 16 runs: max = 14.5s, min = 7.3s, avg = 10.6s, dev = 2.1s //tensorflow/python/data/experimental/kernel_tests/service:dynamic_sharding_test PASSED in 26.5s Stats over 16 runs: max = 26.5s, min = 16.4s, avg = 21.2s, dev = 3.2s //tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test PASSED in 22.0s Stats over 16 runs: max = 22.0s, min = 7.6s, avg = 13.6s, dev = 4.4s //tensorflow/python/data/kernel_tests:snapshot_test PASSED in 23.4s Stats over 16 runs: max = 23.4s, min = 8.5s, avg = 15.0s, dev = 4.0s //tensorflow/python/kernel_tests/control_flow:control_flow_ops_py_test_cpu PASSED in 21.5s Stats over 16 runs: max = 21.5s, min = 4.0s, avg = 7.7s, dev = 3.9s //tensorflow/python/kernel_tests/linalg:matrix_exponential_op_test PASSED in 9.9s Stats over 16 runs: max = 9.9s, min = 3.4s, avg = 5.2s, dev = 2.1s //tensorflow/python/kernel_tests/signal:dct_ops_test_cpu PASSED in 11.6s Stats over 16 runs: max = 11.6s, min = 8.6s, avg = 9.8s, dev = 0.8s //tensorflow/python/ops/parallel_for:control_flow_ops_test_cpu PASSED in 46.9s Stats over 16 runs: max = 46.9s, min = 12.1s, avg = 19.4s, dev = 7.8s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test PASSED in 9.7s Stats over 17 runs: max = 9.7s, min = 3.1s, avg = 5.1s, dev = 1.6s //tensorflow/python/data/kernel_tests:map_test PASSED in 33.1s Stats over 19 runs: max = 33.1s, min = 9.6s, avg = 16.3s, dev = 5.5s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu PASSED in 6.5s Stats over 20 runs: max = 6.5s, min = 3.0s, avg = 4.6s, dev = 1.2s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu_mlir_bridge_test PASSED in 6.5s Stats over 20 runs: max = 6.5s, min = 3.0s, avg = 4.2s, dev = 1.0s //tensorflow/compiler/tests:pooling_ops_test_cpu PASSED in 13.5s Stats over 20 runs: max = 13.5s, min = 5.3s, avg = 7.6s, dev = 1.9s //tensorflow/compiler/tests:pooling_ops_test_cpu_mlir_bridge_test PASSED in 12.1s Stats over 20 runs: max = 12.1s, min = 3.3s, avg = 5.0s, dev = 2.0s //tensorflow/compiler/xla/tests:convolution_dimension_numbers_test_cpu PASSED in 7.7s Stats over 20 runs: max = 7.7s, min = 6.6s, avg = 7.0s, dev = 0.3s //tensorflow/compiler/xla/tests:dot_operation_single_threaded_runtime_test_cpu PASSED in 11.6s Stats over 20 runs: max = 11.6s, min = 9.3s, avg = 10.5s, dev = 0.5s //tensorflow/compiler/xla/tests:dot_operation_test_cpu PASSED in 11.7s Stats over 20 runs: max = 11.7s, min = 10.0s, avg = 10.7s, dev = 0.5s //tensorflow/compiler/xla/tests:prng_test_cpu PASSED in 8.9s Stats over 20 runs: max = 8.9s, min = 7.3s, avg = 8.2s, dev = 0.5s //tensorflow/compiler/xla/tests:reduce_window_test_cpu PASSED in 36.9s Stats over 20 runs: max = 36.9s, min = 5.9s, avg = 14.5s, dev = 9.5s //tensorflow/python/autograph/tests:loop_control_flow_test PASSED in 18.8s Stats over 20 runs: max = 18.8s, min = 10.8s, avg = 14.2s, dev = 2.2s //tensorflow/python/kernel_tests:metrics_test PASSED in 33.8s Stats over 20 runs: max = 33.8s, min = 7.4s, avg = 16.3s, dev = 8.0s //tensorflow/python/kernel_tests/array_ops:matrix_band_part_op_test_cpu PASSED in 7.0s Stats over 20 runs: max = 7.0s, min = 3.1s, avg = 4.8s, dev = 1.3s //tensorflow/python/kernel_tests/data_structures:barrier_ops_test PASSED in 12.2s Stats over 20 runs: max = 12.2s, min = 3.2s, avg = 6.4s, dev = 2.5s //tensorflow/python/kernel_tests/linalg:eig_op_test PASSED in 32.3s Stats over 20 runs: max = 32.3s, min = 3.2s, avg = 11.1s, dev = 9.9s //tensorflow/python/kernel_tests/linalg:linalg_grad_test_cpu PASSED in 92.1s Stats over 20 runs: max = 92.1s, min = 29.4s, avg = 57.7s, dev = 19.8s //tensorflow/python/kernel_tests/linalg:norm_op_test_cpu PASSED in 6.6s Stats over 20 runs: max = 6.6s, min = 3.3s, avg = 4.6s, dev = 1.2s //tensorflow/python/kernel_tests/linalg:normalize_op_test_cpu PASSED in 11.4s Stats over 20 runs: max = 11.4s, min = 4.9s, avg = 7.9s, dev = 1.8s //tensorflow/python/kernel_tests/linalg:qr_op_test_cpu PASSED in 132.8s Stats over 20 runs: max = 132.8s, min = 27.1s, avg = 84.9s, dev = 33.4s //tensorflow/python/kernel_tests/linalg:self_adjoint_eig_op_test_cpu PASSED in 19.2s Stats over 20 runs: max = 19.2s, min = 4.3s, avg = 10.4s, dev = 4.5s //tensorflow/python/kernel_tests/math_ops:batch_matmul_op_test_cpu PASSED in 20.5s Stats over 20 runs: max = 20.5s, min = 5.8s, avg = 12.5s, dev = 4.0s //tensorflow/python/kernel_tests/nn_ops:embedding_ops_test_cpu PASSED in 16.5s Stats over 20 runs: max = 16.5s, min = 9.4s, avg = 11.6s, dev = 1.7s //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu FAILED in 3 out of 22 in 60.1s Stats over 22 runs: max = 60.1s, min = 7.8s, avg = 27.7s, dev = 18.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu/shard_2_of_20/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu/shard_2_of_20/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_cpu/shard_2_of_20/test_attempts/attempt_2.log //tensorflow/python/data/experimental/kernel_tests/service:local_workers_test PASSED in 18.2s Stats over 24 runs: max = 18.2s, min = 4.5s, avg = 11.8s, dev = 3.2s //tensorflow/python/data/kernel_tests:interleave_test PASSED in 22.6s Stats over 24 runs: max = 22.6s, min = 7.9s, avg = 13.5s, dev = 4.2s //tensorflow/python/data/kernel_tests:sample_from_datasets_test PASSED in 17.7s Stats over 24 runs: max = 17.7s, min = 3.3s, avg = 7.8s, dev = 3.6s //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu FAILED in 6 out of 24 in 18.4s Stats over 24 runs: max = 18.4s, min = 12.2s, avg = 15.6s, dev = 1.8s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu/shard_9_of_20/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu/shard_9_of_20/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu/shard_9_of_20/test_attempts/attempt_2.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu/shard_8_of_20/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu/shard_8_of_20/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/kernel_tests/math_ops/matmul_op_test_cpu/shard_8_of_20/test_attempts/attempt_2.log //tensorflow/compiler/xla/tests:array_elementwise_ops_test_cpu PASSED in 7.5s Stats over 25 runs: max = 7.5s, min = 6.6s, avg = 7.0s, dev = 0.3s //tensorflow/compiler/xla/tests:select_and_scatter_test_cpu PASSED in 34.2s Stats over 25 runs: max = 34.2s, min = 6.8s, avg = 12.1s, dev = 6.9s //tensorflow/compiler/xla/tests:convolution_variants_test_cpu PASSED in 7.4s Stats over 30 runs: max = 7.4s, min = 6.6s, avg = 7.0s, dev = 0.3s //tensorflow/compiler/xla/tests:iota_test_cpu PASSED in 14.0s Stats over 30 runs: max = 14.0s, min = 12.4s, avg = 13.3s, dev = 0.4s //tensorflow/compiler/xla/tests:params_test_cpu PASSED in 9.9s Stats over 30 runs: max = 9.9s, min = 6.9s, avg = 8.0s, dev = 0.8s //tensorflow/compiler/xla/tests:reshape_test_cpu PASSED in 11.5s Stats over 30 runs: max = 11.5s, min = 6.3s, avg = 7.7s, dev = 1.0s //tensorflow/python/kernel_tests/nn_ops:conv_ops_3d_test_cpu PASSED in 27.2s Stats over 30 runs: max = 27.2s, min = 2.9s, avg = 9.5s, dev = 6.7s //tensorflow/compiler/xla/tests:reduce_test_cpu PASSED in 9.3s Stats over 31 runs: max = 9.3s, min = 7.3s, avg = 8.1s, dev = 0.6s //tensorflow/compiler/xla/tests:scalar_computations_test_cpu PASSED in 9.4s Stats over 32 runs: max = 9.4s, min = 6.8s, avg = 7.5s, dev = 0.6s //tensorflow/python/data/experimental/kernel_tests/service:auto_shard_test PASSED in 17.1s Stats over 32 runs: max = 17.1s, min = 2.9s, avg = 10.5s, dev = 3.3s //tensorflow/python/data/experimental/kernel_tests/service:data_service_ops_test PASSED in 27.2s Stats over 32 runs: max = 27.2s, min = 6.7s, avg = 13.1s, dev = 4.6s //tensorflow/compiler/xla/tests:batch_normalization_test_cpu PASSED in 9.8s Stats over 40 runs: max = 9.8s, min = 7.4s, avg = 8.5s, dev = 0.6s //tensorflow/compiler/xla/tests:bfloat16_test_cpu PASSED in 8.9s Stats over 40 runs: max = 8.9s, min = 6.8s, avg = 7.9s, dev = 0.7s //tensorflow/compiler/xla/tests:conv_depthwise_backprop_filter_test_cpu PASSED in 9.8s Stats over 40 runs: max = 9.8s, min = 7.6s, avg = 8.6s, dev = 0.6s //tensorflow/compiler/xla/tests:slice_test_cpu PASSED in 10.4s Stats over 40 runs: max = 10.4s, min = 7.2s, avg = 8.4s, dev = 0.7s //tensorflow/compiler/mlir/quantization/tensorflow/python:quantize_model_test PASSED in 39.3s Stats over 50 runs: max = 39.3s, min = 17.8s, avg = 25.8s, dev = 5.2s //tensorflow/compiler/tests:sort_ops_test_cpu PASSED in 34.9s Stats over 50 runs: max = 34.9s, min = 2.7s, avg = 9.2s, dev = 6.8s //tensorflow/compiler/tests:sort_ops_test_cpu_mlir_bridge_test PASSED in 40.8s Stats over 50 runs: max = 40.8s, min = 2.8s, avg = 11.6s, dev = 8.5s //tensorflow/compiler/xla/tests:conv_depthwise_test_cpu PASSED in 11.1s Stats over 50 runs: max = 11.1s, min = 7.3s, avg = 8.7s, dev = 0.8s //tensorflow/compiler/xla/tests:convolution_test_1d_no_vmodule_cpu PASSED in 10.8s Stats over 50 runs: max = 10.8s, min = 8.2s, avg = 9.3s, dev = 0.8s //tensorflow/compiler/xla/tests:convolution_test_cpu PASSED in 16.8s Stats over 50 runs: max = 16.8s, min = 9.7s, avg = 12.5s, dev = 1.2s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_grad_test_cpu PASSED in 13.2s Stats over 50 runs: max = 13.2s, min = 4.2s, avg = 7.7s, dev = 2.4s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_grad_test_cpu PASSED in 6.7s Stats over 50 runs: max = 6.7s, min = 3.0s, avg = 4.9s, dev = 1.2s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_sparse_mat_mul_grad_test_cpu PASSED in 6.8s Stats over 50 runs: max = 6.8s, min = 2.9s, avg = 4.2s, dev = 1.3s //tensorflow/python/kernel_tests/math_ops:cwise_ops_binary_test_cpu PASSED in 27.4s Stats over 50 runs: max = 27.4s, min = 6.7s, avg = 13.6s, dev = 5.4s //tensorflow/python/kernel_tests/math_ops:cwise_ops_test_cpu PASSED in 12.5s Stats over 50 runs: max = 12.5s, min = 2.9s, avg = 4.0s, dev = 1.6s //tensorflow/python/kernel_tests/math_ops:cwise_ops_unary_test_cpu PASSED in 11.5s Stats over 50 runs: max = 11.5s, min = 2.8s, avg = 4.2s, dev = 2.0s Executed 3771 out of 3771 tests: 3768 tests pass and 3 fail locally. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.