Skip to content

test_ssl: test_get_server_certificate() and test_msg_callback_deadlock_bpo43577() fail randomly on the macOS CI #88395

Closed
Closed
test_ssl: test_get_server_certificate() and test_msg_callback_deadlock_bpo43577() fail randomly on the macOS CI#88395
Labels
3.10 (EOL)end of life3.11only security fixes3.8 (EOL)end of life3.9 (EOL)end of lifetestsTests in the Lib/test dir

Description

@erlend-aasland
erlend-aasland
opened on May 25, 2021
BPO 44229
Nosy @terryjreedy, @ronaldoussoren, @vstinner, @tiran, @pablogsal, @miss-islington, @erlend-aasland
PRs
  • bpo-44229: Intest_ssl, ignore spurious EPROTOTYPE on macOS #26893
  • [3.9] bpo-44229: Ignore spurious EPROTOTYPE on macOS in test_ssl (GH-26893) #26894
  • [3.8] bpo-44229: Ignore spurious EPROTOTYPE on macOS in test_ssl (GH-26893) #26895
  • [3.10] bpo-44229: Ignore spurious EPROTOTYPE on macOS in test_ssl (GH-26893) #26896
    		•
    Files
  • 5_Display build info.txt
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-06-27.14:17:20.638>
created_at = <Date 2021-05-25.07:24:56.023>
labels = ['3.8', 'tests', '3.9', '3.10', '3.11']
title = 'test_ssl: test_get_server_certificate() and test_msg_callback_deadlock_bpo43577() fail randomly on the macOS CI'
updated_at = <Date 2021-06-28.08:11:11.070>
user = 'https://github.com/erlend-aasland'

    bugs.python.org fields:

    activity = <Date 2021-06-28.08:11:11.070>
actor = 'erlendaasland'
assignee = 'none'
closed = True
closed_date = <Date 2021-06-27.14:17:20.638>
closer = 'erlendaasland'
components = ['Tests']
creation = <Date 2021-05-25.07:24:56.023>
creator = 'erlendaasland'
dependencies = []
files = ['50063']
hgrepos = []
issue_num = 44229
keywords = ['patch']
message_count = 40.0
messages = ['394299', '394321', '394339', '394341', '394342', '394344', '394347', '394348', '394349', '394355', '394357', '394367', '394418', '394420', '394739', '394741', '394751', '394752', '394798', '394805', '396149', '396425', '396431', '396436', '396438', '396441', '396464', '396466', '396471', '396474', '396479', '396481', '396482', '396485', '396487', '396490', '396580', '396613', '396615', '396616']
nosy_count = 7.0
nosy_names = ['terry.reedy', 'ronaldoussoren', 'vstinner', 'christian.heimes', 'pablogsal', 'miss-islington', 'erlendaasland']
pr_nums = ['26893', '26894', '26895', '26896']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue44229'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

    Activity

    1. erlend-aasland commented on May 25, 2021

      @erlend-aasland
      erlend-aasland
      on May 25, 2021
      ContributorAuthor

      See https://github.com/python/cpython/pull/26104/checks?check_run_id=2662511684

      ======================================================================
      ERROR: test_get_server_certificate (test.test_ssl.SimpleBackgroundTests)
      ----------------------------------------------------------------------

      Traceback (most recent call last):
  File "/Users/runner/work/cpython/cpython/Lib/test/test_ssl.py", line 2132, in test_get_server_certificate
    _test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
  File "/Users/runner/work/cpython/cpython/Lib/test/test_ssl.py", line 2329, in _test_get_server_certificate
    pem = ssl.get_server_certificate((host, port), ca_certs=cert)
  File "/Users/runner/work/cpython/cpython/Lib/ssl.py", line 1520, in get_server_certificate
    with create_connection(addr, timeout=timeout) as sock:
  File "/Users/runner/work/cpython/cpython/Lib/socket.py", line 844, in create_connection
    raise err
  File "/Users/runner/work/cpython/cpython/Lib/socket.py", line 832, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
    2. erlend-aasland
      added
      3.11only security fixes
      testsTests in the Lib/test dir
      on May 25, 2021
    3. erlend-aasland
      changed the title [-]test_get_server_certificate fails on macOS[/-] [+]test_get_server_certificate fails intermittently on macOS[/+] on May 25, 2021
    4. erlend-aasland
      changed the title [-]test_get_server_certificate fails on macOS[/-] [+]test_get_server_certificate fails intermittently on macOS[/+] on May 25, 2021

    5. tiran commented on May 25, 2021

      @tiran
      tiran
      on May 25, 2021
      Member

      The problem could be related to bpo-43921. I neither have a macOS nor a Windows machine to reproduce and debug the issue. Since I'm cannot reproduce the problem on Linux, I'm unable to debug and fix it.

    6. pablogsal commented on May 25, 2021

      @pablogsal
      pablogsal
      on May 25, 2021
      Member

      I neither have a macOS nor a Windows machine to reproduce and debug the issue.

      Can you maybe use a VM for Windows?

    7. erlend-aasland commented on May 25, 2021

      @erlend-aasland
      erlend-aasland
      on May 25, 2021
      ContributorAuthor

      I’ll see if I can provoke it on my Mac.

      What’s the preferred solution?

      1. Catch the connection failure and skip the test?
      2. Retry connection?
      3. Find the root cause and make sure it never ever happens :)

      Maybe 2. is acceptable :)

    8. pablogsal commented on May 25, 2021

      @pablogsal
      pablogsal
      on May 25, 2021
      Member

      Is possible that these are flaky tests but I am afraid this may be something more important so o would prefer to go with 3 ;)

    9. erlend-aasland commented on May 25, 2021

      @erlend-aasland
      erlend-aasland
      on May 25, 2021
      ContributorAuthor

      I feared that ;)

    10. erlend-aasland commented on May 25, 2021

      @erlend-aasland
      erlend-aasland
      on May 25, 2021
      ContributorAuthor

      Getting there. The error message is similar:

      $ ./python.exe -m test test_ssl -m test_get_server_certificate -u all -F
...
0:02:11 load avg: 2.10 [328] test_ssl
test test_ssl failed -- Traceback (most recent call last):
  File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2132, in test_get_server_certificate
    _test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
  File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2329, in _test_get_server_certificate
    pem = ssl.get_server_certificate((host, port), ca_certs=cert)
  File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1521, in get_server_certificate
    with context.wrap_socket(sock, server_hostname=host) as sslsock:
  File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 518, in wrap_socket
    return self.sslsocket_class._create(
  File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1070, in _create
    self.do_handshake()
  File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1339, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 54] Connection reset by peer

      test_ssl failed

      == Tests result: FAILURE ==

    11. erlend-aasland commented on May 25, 2021

      @erlend-aasland
      erlend-aasland
      on May 25, 2021
      ContributorAuthor

      Got it:

      0:02:29 load avg: 2.81 [389] test_ssl
test test_ssl failed -- Traceback (most recent call last):
  File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2132, in test_get_server_certificate
    _test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
  File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2329, in _test_get_server_certificate
    pem = ssl.get_server_certificate((host, port), ca_certs=cert)
  File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1520, in get_server_certificate
    with create_connection(addr, timeout=timeout) as sock:
  File "/Users/erlendaasland/src/cpython-ssl/Lib/socket.py", line 844, in create_connection
    raise err
  File "/Users/erlendaasland/src/cpython-ssl/Lib/socket.py", line 832, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

      test_ssl failed

      == Tests result: FAILURE ==

    12. erlend-aasland commented on May 25, 2021

      @erlend-aasland
      erlend-aasland
      on May 25, 2021
      ContributorAuthor
      $ git diff
diff --git a/Lib/ssl.py b/Lib/ssl.py
index 2b131de043..9c281d8028 100644
--- a/Lib/ssl.py
+++ b/Lib/ssl.py
@@ -257,8 +257,9 @@ class _TLSMessageType:
 if sys.platform == "win32":
     from _ssl import enum_certificates, enum_crls
 
+from test.support import SHORT_TIMEOUT as _GLOBAL_DEFAULT_TIMEOUT
 from socket import socket, SOCK_STREAM, create_connection
-from socket import SOL_SOCKET, SO_TYPE, _GLOBAL_DEFAULT_TIMEOUT
+from socket import SOL_SOCKET, SO_TYPE
 import socket as _socket
 import base64        # for DER-to-PEM translation
 import errno
      $ % ./python.exe -m test test_ssl -m test_get_server_certificate -u all -F
...
0:03:27 load avg: 2.24 [535] test_ssl
test test_ssl failed -- Traceback (most recent call last):
  File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2132, in test_get_server_certificate
    _test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
  File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2329, in _test_get_server_certificate
    pem = ssl.get_server_certificate((host, port), ca_certs=cert)
  File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1521, in get_server_certificate
    with create_connection(addr, timeout=timeout) as sock:
  File "/Users/erlendaasland/src/cpython-ssl/Lib/socket.py", line 844, in create_connection
    raise err
  File "/Users/erlendaasland/src/cpython-ssl/Lib/socket.py", line 832, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

      test_ssl failed

      == Tests result: FAILURE ==

    13. 29 remaining items

    14. erlend-aasland
      added
      3.10 (EOL)end of life
      on Jun 24, 2021

    15. pablogsal commented on Jun 24, 2021

      @pablogsal
      pablogsal
      on Jun 24, 2021
      Member

      New changeset b5a52ee by Erlend Egeberg Aasland in branch 'main':
      bpo-44229: Ignore spurious EPROTOTYPE on macOS in test_ssl (GH-26893)
      b5a52ee

    16. miss-islington commented on Jun 24, 2021

      @miss-islington
      miss-islington
      on Jun 24, 2021
      Contributor

      New changeset 0796e21 by Miss Islington (bot) in branch '3.9':
      bpo-44229: Ignore spurious EPROTOTYPE on macOS in test_ssl (GH-26893)
      0796e21

    17. miss-islington commented on Jun 24, 2021

      @miss-islington
      miss-islington
      on Jun 24, 2021
      Contributor

      New changeset b3fac29 by Miss Islington (bot) in branch '3.10':
      bpo-44229: Ignore spurious EPROTOTYPE on macOS in test_ssl (GH-26893)
      b3fac29

    18. erlend-aasland commented on Jun 24, 2021

      @erlend-aasland
      erlend-aasland
      on Jun 24, 2021
      ContributorAuthor

      Marking gh-88021 (bpo-43855) as a duplicate of this issue.

    19. erlend-aasland commented on Jun 24, 2021

      @erlend-aasland
      erlend-aasland
      on Jun 24, 2021
      ContributorAuthor

      Marking gh-88403 (bpo-44237) as a duplicate of this issue.

    20. erlend-aasland commented on Jun 24, 2021

      @erlend-aasland
      erlend-aasland
      on Jun 24, 2021
      ContributorAuthor

      FYI, here's a CI failure against 3.9:
      https://github.com/python/cpython/runs/2894813367?check_suite_focus=true

    21. pablogsal commented on Jun 24, 2021

      @pablogsal
      pablogsal
      on Jun 24, 2021
      Member

      New changeset 71ba16b by Miss Islington (bot) in branch '3.8':
      bpo-44229: Ignore spurious EPROTOTYPE on macOS in test_ssl (GH-26893) (GH-26895)
      71ba16b

    22. erlend-aasland commented on Jun 27, 2021

      @erlend-aasland
      erlend-aasland
      on Jun 27, 2021
      ContributorAuthor

      I haven't observed this issue on the CI since GH-26893 through GH-26896 was merged. Marking this as resolved. If anyone disagrees, feel free to reopen :)

    23. erlend-aasland
      closed this as completedon Jun 27, 2021
    24. erlend-aasland
      closed this as completedon Jun 27, 2021

    25. vstinner commented on Jun 28, 2021

      @vstinner
      vstinner
      on Jun 28, 2021
      Member

      http://erickt.github.io/blog/2014/11/19/adventures-in-debugging-a-potential-osx-kernel-bug/ says:

      "If we trigger a send while the kernel is in the middle of tearing down the socket, it returns EPROTOTYPE." (instead of ECONNRESET)

      Maybe send() could raise a ConnectionResetError exception on EPROTOTYPE?

    26. erlend-aasland commented on Jun 28, 2021

      @erlend-aasland
      erlend-aasland
      on Jun 28, 2021
      ContributorAuthor

      But here, it is write() that returns EPROTOTYPE. See msg394798.

    27. erlend-aasland commented on Jun 28, 2021

      @erlend-aasland
      erlend-aasland
      on Jun 28, 2021
      ContributorAuthor

      I'd be interested in hearing Ronald's opinion. (Added to nosy.)

    28. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.10 (EOL)end of life3.11only security fixes3.8 (EOL)end of life3.9 (EOL)end of lifetestsTests in the Lib/test dir

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions