Add full FinTS parser.

[henryk]

Significantly changes the internal API, as segments are now lists of lists.

Fully supports all escaping and binary content. Decodes 'normal' data from ISO-8859-1 to Python strings, leaves binary data as binary.

Still broken: get_holdings()/HIWPD

In the end I want a proper object based internal representation that can be parsed from and serialized into the network, with classes and fields like Django models. For the time being this only touches the receiving code and simplifies all the instances of split_for_data_groups() and split_for_data_elements().

The handling of FinTS security (segment HNVSD) is a bit hacky: The contents of HNVSD are separately parsed and stored in FinTSResponse::payload, and _find_segments will first search the outer message and then the inner message, to satisfy the existing code.

[raphaelm]

Sounds great! I will probably not have time to look at this in detail this week, but at a quick glance it looks like a good direction for this project.

[henryk]

Ok, so I did a little work, and, uhhm, this might need to drop the word "minimal" in the project name. ;)

I'm still fiddling with code structure and what should go into which package, so it's not pushed here yet, experimenting in https://github.com/henryk/python-fints/tree/parser-ng. Also: tests, with ~80% coverage of the new code.

In [1]: from fints.formals import *

In [2]: import fints.segments

In [3]: from fints.parser import FinTS3Parser

In [4]: data = \
   ...: (b'HNHBK:1:3+000000000428+300+430711670077=043999659571CN9D=+2+430711670077=043'
   ...:  b"999659571CN9D=:2'HNVSK:998:3+PIN:1+998+1+2::oIm3BlHv6mQBAADYgbPpp?+kWrAQA+1+"
   ...:  b"2:2:13:@8@00000000:5:1+280:15050500:hermes:S:0:0+0'HNVSD:999:1+@195@HNSHK:2:"
   ...:  b'4+PIN:1+999+9166926+1+1+2::oIm3BlHv6mQBAADYgbPpp?+kWrAQA+1+1+1:999:1+6:10:16'
   ...:  b"+280:15050500:hermes:S:0:0'HIRMG:3:2+0010::Nachricht entgegengenommen.+0100:"
   ...:  b":Dialog beendet.'HNSHA:4:2+9166926''HNHBS:5:1+2'")
   ...:

In [5]: m = FinTS3Parser().parse_message(data)

In [6]: m.print_nested()

SegmentSequence([
    fints.segments.HNHBK3(
        header = fints.formals.SegmentHeader(
                type = 'HNHBK',
                number = 1,
                version = 3,
                reference = None,
            ),
        message_size = '000000000428',
        hbci_version = 300,
        dialogue_id = '430711670077=043999659571CN9D=',
        message_number = 2,
        reference_message = fints.formals.ReferenceMessage(
                dialogue_id = '430711670077=043999659571CN9D=',
                message_number = 2,
            ),
    ),
    fints.segments.HNVSK3(
        header = fints.formals.SegmentHeader(
                type = 'HNVSK',
                number = 998,
                version = 3,
                reference = None,
            ),
        security_profile = fints.formals.SecurityProfile(
                security_method = 'PIN',
                security_method_version = 1,
            ),
        security_function = '998',
        security_role = '1',
        security_identification_details = fints.formals.SecurityIdentificationDetails(
                name_party = '2',
                cid = None,
                identifier_party = 'oIm3BlHv6mQBAADYgbPpp+kWrAQA',
            ),
        security_datetime = fints.formals.SecurityDateTime(
                datetime_type = '1',
                date = None,
                time = None,
            ),
        encryption_algorithm = fints.formals.EncryptionAlgorithm(
                usage_encryption = '2',
                operation_mode = '2',
                encryption_algorithm = '13',
                algorithm_parameter_value = b'00000000',
                algorithm_parameter_name = '5',
                algorithm_parameter_iv_name = '1',
                algorithm_parameter_iv_value = None,
            ),
        key_name = fints.formals.KeyName(
                bank_identifier = fints.formals.BankIdentifier(
                        country_identifier = '280',
                        bank_code = '15050500',
                    ),
                user_id = 'hermes',
                key_type = 'S',
                key_number = 0,
                key_version = 0,
            ),
        compression_function = '0',
        certificate = fints.formals.Certificate(
                certificate_type = None,
                certificate_content = None,
            ),
    ),
    fints.segments.HNVSD1(
        header = fints.formals.SegmentHeader(
                type = 'HNVSD',
                number = 999,
                version = 1,
                reference = None,
            ),
        data = SegmentSequence([
                fints.segments.HNSHK4(
                    header = fints.formals.SegmentHeader(
                            type = 'HNSHK',
                            number = 2,
                            version = 4,
                            reference = None,
                        ),
                    security_profile = fints.formals.SecurityProfile(
                            security_method = 'PIN',
                            security_method_version = 1,
                        ),
                    security_function = '999',
                    security_reference = '9166926',
                    security_application_area = '1',
                    security_role = '1',
                    security_identification_details = fints.formals.SecurityIdentificationDetails(
                            name_party = '2',
                            cid = None,
                            identifier_party = 'oIm3BlHv6mQBAADYgbPpp+kWrAQA',
                        ),
                    security_reference_number = 1,
                    security_datetime = fints.formals.SecurityDateTime(
                            datetime_type = '1',
                            date = None,
                            time = None,
                        ),
                    hash_algorithm = fints.formals.HashAlgorithm(
                            usage_hash = '1',
                            hash_algorithm = '999',
                            algorithm_parameter_name = '1',
                            algorithm_parameter_value = None,
                        ),
                    signature_algorithm = fints.formals.SignatureAlgorithm(
                            usage_signature = '6',
                            signature_algorithm = '10',
                            operation_mode = '16',
                        ),
                    key_name = fints.formals.KeyName(
                            bank_identifier = fints.formals.BankIdentifier(
                                    country_identifier = '280',
                                    bank_code = '15050500',
                                ),
                            user_id = 'hermes',
                            key_type = 'S',
                            key_number = 0,
                            key_version = 0,
                        ),
                    certificate = fints.formals.Certificate(
                            certificate_type = None,
                            certificate_content = None,
                        ),
                ),
                fints.segments.FinTS3Segment(
                    header = fints.formals.SegmentHeader(
                            type = 'HIRMG',
                            number = 3,
                            version = 2,
                            reference = None,
                        ),
                    _additional_data=
                        [['0010', None, 'Nachricht entgegengenommen.'], ['0100', None, 'Dialog beendet.']],
                ),
                fints.segments.FinTS3Segment(
                    header = fints.formals.SegmentHeader(
                            type = 'HNSHA',
                            number = 4,
                            version = 2,
                            reference = None,
                        ),
                    _additional_data=
                        ['9166926'],
                ),
            ]),
    ),
    fints.segments.HNHBS1(
        header = fints.formals.SegmentHeader(
                type = 'HNHBS',
                number = 5,
                version = 1,
                reference = None,
            ),
        message_number = 2,
    ),
])

In [7]: m2 = SegmentSequence([ ..... Everything that was output after In [6] .... ])

In [8]: m2
Out[8]: SegmentSequence([fints.segments.HNHBK3(header=fints.formals.SegmentHeader(type='HNHBK', number=1, version=3, reference=None), message_size='000000000428', hbci_version=300, dialogue_id='430711670077=043999659571CN9D=', message_number=2, reference_message=fints.formals.ReferenceMessage(dialogue_id='430711670077=043999659571CN9D=', message_number=2)), fints.segments.HNVSK3(header=fints.formals.SegmentHeader(type='HNVSK', number=998, version=3, reference=None), security_profile=fints.formals.SecurityProfile(security_method='PIN', security_method_version=1), security_function='998', security_role='1', security_identification_details=fints.formals.SecurityIdentificationDetails(name_party='2', cid=None, identifier_party='oIm3BlHv6mQBAADYgbPpp+kWrAQA'), security_datetime=fints.formals.SecurityDateTime(datetime_type='1', date=None, time=None), encryption_algorithm=fints.formals.EncryptionAlgorithm(usage_encryption='2', operation_mode='2', encryption_algorithm='13', algorithm_parameter_value=b'00000000', algorithm_parameter_name='5', algorithm_parameter_iv_name='1', algorithm_parameter_iv_value=None), key_name=fints.formals.KeyName(bank_identifier=fints.formals.BankIdentifier(country_identifier='280', bank_code='15050500'), user_id='hermes', key_type='S', key_number=0, key_version=0), compression_function='0', certificate=fints.formals.Certificate(certificate_type=None, certificate_content=None)), fints.segments.HNVSD1(header=fints.formals.SegmentHeader(type='HNVSD', number=999, version=1, reference=None), data=SegmentSequence([fints.segments.HNSHK4(header=fints.formals.SegmentHeader(type='HNSHK', number=2, version=4, reference=None), security_profile=fints.formals.SecurityProfile(security_method='PIN', security_method_version=1), security_function='999', security_reference='9166926', security_application_area='1', security_role='1', security_identification_details=fints.formals.SecurityIdentificationDetails(name_party='2', cid=None, identifier_party='oIm3BlHv6mQBAADYgbPpp+kWrAQA'), security_reference_number=1, security_datetime=fints.formals.SecurityDateTime(datetime_type='1', date=None, time=None), hash_algorithm=fints.formals.HashAlgorithm(usage_hash='1', hash_algorithm='999', algorithm_parameter_name='1', algorithm_parameter_value=None), signature_algorithm=fints.formals.SignatureAlgorithm(usage_signature='6', signature_algorithm='10', operation_mode='16'), key_name=fints.formals.KeyName(bank_identifier=fints.formals.BankIdentifier(country_identifier='280', bank_code='15050500'), user_id='hermes', key_type='S', key_number=0, key_version=0), certificate=fints.formals.Certificate(certificate_type=None, certificate_content=None)), fints.segments.FinTS3Segment(header=fints.formals.SegmentHeader(type='HIRMG', number=3, version=2, reference=None), _additional_data=[['0010', None, 'Nachricht entgegengenommen.'], ['0100', None, 'Dialog beendet.']]), fints.segments.FinTS3Segment(header=fints.formals.SegmentHeader(type='HNSHA', number=4, version=2, reference=None), _additional_data=['9166926'])])), fints.segments.HNHBS1(header=fints.formals.SegmentHeader(type='HNHBS', number=5, version=1, reference=None), message_number=2)])

In [9]: m2.segments[2].data.segments[0].security_profile
Out[9]: fints.formals.SecurityProfile(security_method='PIN', security_method_version=1)

[raphaelm]

Hi!

I haven't looked at the implementation in detail yet, just at the output and at the formals module, and I love it ❤️

Please let me know if there are any specific design decisions you'd like to have feedback on!

[henryk]

Ok, all the core work is done. I've successfully retrieved an account list (not quite a statement yet ;) with the new code in the parser path.
There is a debug print in connection.py to pretty-print (and test the parser) all incoming and outgoing traffic. (The PasswordField class has fints.utils.Password as its type, so that works seamlessly :)
(Also, f you put a raw binary message into tests/messages/private_something.bin you can use it to test the parser and see its output: pytest tests/ -k "test_parse_other[private_something]" -s. The private_*.bin files are ignored by .git)

Currently I haven't touched the generating/sending bit (and renamed FinTS3Segment to FinTS3SegmentOLD to keep the code running), I'm still slightly unclear as to how best do the "encryption" envelope thing, and segment numbers. (The current code hardcodes segment numbers at segment constructor call time, that should ideally be automatically done by the message class.)

I'm also somewhat unhappy with the property names. I've loosely translated most of them just because I needed them to be there to go on. Once this API is in use, it's hard to change the property names, so they should be reviewed and improved before that.

What needs to be done (and I don't want to do alone):

• Somehow usefully separate out all the classes into different modules, perhaps? Also with regards to automodule for Sphinx (yes, there's documentation :)
• Review, improve, fix property names. Make sure that all list fields have a plural name (HIRMG2.responses) and all single fields have singular names (HITANS6.parameter).
• Implement the missing classes/merge over the existing fints.segments classes
• Implement a useful FinTSMessage class that automatically handles nesting/enveloping and segment numbers
• Fix everything that I've broken in the TAN mechanism stuff. Should be easier now ;)

What would be nice:

• A repository of test cases, without private data in them
• Test accounts to test online

[raphaelm]

This is amazing! Unfortunately, I lack the time to help at the moment since I'm to deeply involved with other problems and will be on vacation August 22–30th, but I'll be happy to help or take over at some point if I have more time on my hands.

[henryk]

Getting closer, first working transfer sent.

I have two new secondary directives when designing the API:

1. State can be saved and restored. Goal: Make it usable from a web platform where objects don't persist between page views. The bank can require that a TAN is sent in the same dialogue as the original message, so we need to be able to resume dialogues.
2. Make the API so that future extensions don't break it. Specifically: Allow for the inclusion of other security mechanisms (e.g. HBCI, one-step TAN) that don't require two steps.

ad 1) Looks like this:

client = FinTS3PinTanClient(..., set_data=None)
with client:
    response = client.start_sepa_transfer(...)
    dialog_data = client.pause_dialog()

challenge_data = response.get_data()
client_data = client.get_data()

# Store challenge_data, dialog_data and client_data out-of-band somewhere
# Ask the user to respond to response.hitan.challenge
# ... Some time passes ...
# Later, possibly in a different process, restore the state

client = FinTS3PinTanClient(..., set_data=client_data)
challenge = NeedRetryResponse.from_data(challenge_data)
with client.resume_dialog(dialog_data):
    client.send_tan(challenge, tan)

ad 2) Methods like start_sepa_transfer() return either the final response value (not implemented yet :), or an instance of (a subclass of) NeedRetryResponse. Kind of like a future, it will store both the current state and the next step to execute. In the PIN/TAN case: if you get an instance of NeedTANResponse, you need to ask the user for the TAN, then call .send_tan() which will complete the original start_sepa_transfer() call.

The developer point of view is:

def start_sepa_transfer(...):
        return self._send_with_possible_retry(dialog, seg, self._continue_start_sepa_transfer)

def _continue_start_sepa_transfer(self, command_seg, response):
        # FIXME Properly find return code
        return True

where _send_with_possible_retry(self, dialog, command_seg, resume_func) will either directly call its resume_func argument (if no second step is necessary) or leave that up to .send_tan().

As of now I've not done HKTAN#6 on purpose since that implies Strong Customer Authentication (SCA) which comes with its own set of cans of worms (including TANs for dialogue initiation/"login", TAN exemptions for certain commands based on bank decisions).

[henryk]

Side note: I've established an anonymous dialogue with most banks in Germany, 2045 unique BLZ/URL combinations, to test the parser. For future reference here is the statistics of supported HITANS versions:

Counter({(5,): 1172, (1, 3): 399, (4, 5): 284, (): 141, (1, 4, 5): 15, (2,): 8, (2, 5): 7, (1,): 5, (1, 2, 3, 4, 5): 5, (2, 3): 4, (1, 2): 2, (2, 4, 5): 2, (1, 2, 3, 4): 1})

(e.g. 1172 banks support only HITAN#5, 399 banks support HITAN#1 and HITAN#3, etc.)

[henryk]

I'm mostly content with functionality now (and, in parallel, building a byro-fints plugin to use it). Still not everything cleaned up, not all FIXMEs removed. Goal: Test coverage also for the client (with mocked server).

I've broken the API on purpose in most places, so this should get a new major version number.

• start_sepa_debit, start_sepa_transfer is now just sepa_debit/sepa_transfer and return either a NeedTANResponse or TransactionResponse.
• get_statement is renamed to get_transactions because that's what it does. I intend on building a get_statement in the future.

Question for @raphaelm: I'm now working on the TAN part, formatting of the challenge. I'd like to pull in https://pypi.org/project/bleach/ as a dependency to clean-up "challenge_structured = True" challenges. (I can do this in byro-fints, but maybe it's better in the library for everyone to use.)

[raphaelm]

+1 for bleach as a dependency. I'm still (now back from vacation, but with an event coming up in a few days) struggling to find an afternoon to invest in this, please bear with me for another while.

[raphaelm]

(also +1 for breaking the API if it gets better that way)

[raphaelm]

Hi! :) I did a first rough review of this, mostly of the documentation to get an idea of the design. So far, I love it a lot. I attached 2 small design questions that I'd like to hear your opinion about.

I plan to dive deeper into the code next week, but will probably merge it more or less as-is, test around and then go from there. Among the FIXMEs, is there any one that you think we should absolutely tacke before that? I could try to resolve them myself.

[henryk]

As for the FIXMEs: The one in client.py:process_response_message() is basically not yet implemented functionality (a.k.a a bug). I was too lazy to properly implement searching the history for the segment that the response refers to, but have written that into the API, because some library users might need it (to automatically process responses). It probably should be either fixed or documented as a current limitation.

The one in dialog.py:send() is more of a note to self. Previously the assert was a sanity check on received/expected message sequence numbers (and prevents message processing out of order). However, an exception during processing would exit the context handler, would close the dialog, would send() a HKEND message, which would then seem out of order. I'm not sure how to properly[tm] address it: Find the right invariant that is reentrancy safe, or ignore HKEND messages, or skip checks in exception context, or something else.

In formals.py: The original SEPAAccount object is missing a country identifier and implicitly limited to Germany. I wonder whether we'd want to, since we're breaking the API anyway, augment or replace it.

The names for StatusSEPATask1 enum members: If you accept the ones I made up, just delete the FIXME. (Problem is that many things in the spec only have German prose descriptions, but enum members need short identifiers.)

[raphaelm]

FYI, I just tried using this branch for our actual monthly SEPA batch debit. I committed a few small fixes, but now it works.

In formals.py: The original SEPAAccount object is missing a country identifier and implicitly limited to Germany. I wonder whether we'd want to, since we're breaking the API anyway, augment or replace it.

I wouldn't worry too much about extending that tuple (since it's unlikely anyone creates it manually right now), but do we need to? The IBAN does contain country information.

[raphaelm]

TODO for myself to get this done:

• Review and refine documentation
• Documentation for users upgrading from 1.0
• Resolve other discussions in this PR
• Review and improve naming of properties
• Address sequence numbers
• Think about country identifier
• Test reading operations with all banks I have access to
• Test an actual debit
• Test an actual "simple" transfer
• Test an actual transfer
• Add travis setup

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (master@a598030). Click here to learn what that means.
The diff coverage is 92.13%.

@@            Coverage Diff            @@
##             master      #34   +/-   ##
=========================================
  Coverage          ?   89.43%           
=========================================
  Files             ?       23           
  Lines             ?     2933           
  Branches          ?        0           
=========================================
  Hits              ?     2623           
  Misses            ?      310           
  Partials          ?        0

Impacted Files	Coverage Δ
fints/models.py	100% <ø> (ø)
fints/segments/base.py	100% <100%> (ø)
fints/hhd/flicker.py	78.64% <100%> (ø)
fints/segments/transfer.py	100% <100%> (ø)
fints/segments/depot.py	100% <100%> (ø)
fints/segments/saldo.py	100% <100%> (ø)
fints/exceptions.py	100% <100%> (ø)
fints/segments/auth.py	100% <100%> (ø)
fints/segments/message.py	100% <100%> (ø)
fints/connection.py	96.15% <100%> (ø)
... and 13 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a598030...c416e55. Read the comment docs.

[raphaelm]

Side note: I've established an anonymous dialogue with most banks in Germany, 2045 unique BLZ/URL combinations, to test the parser. For future reference here is the statistics of supported HITANS versions:

Counter({(5,): 1172, (1, 3): 399, (4, 5): 284, (): 141, (1, 4, 5): 15, (2,): 8, (2, 5): 7, (1,): 5, (1, 2, 3, 4, 5): 5, (2, 3): 4, (1, 2): 2, (2, 4, 5): 2, (1, 2, 3, 4): 1})

(e.g. 1172 banks support only HITAN#5, 399 banks support HITAN#1 and HITAN#3, etc.)

Do you have that script still around? Would probably be useful to have it somewhere to re-try this later, and if it's just to use it as a test for parser changes.

[raphaelm]

Okay, before this stays around forever, let's merge this. My todo is over, my tests are passing. Holdings probably won't work, but maybe @gonium wants to test that and report back? I still don't have access to any.