Hudson River Trading’s (HRT’s) Python codebase is large* and constantly evolving. Millions of lines of Python reflect the work of hundreds of developers over the last decade. We trade in over 200 markets worldwide — including nearly all of the world’s electronic markets — so we need to regularly update our code to handle changing rules and regulations.

* To learn how we handle large-scale code organization, see: Building a Dependency Graph of Our Python Codebase.

Our codebase provides command-line interface (CLI) tools, graphical user interfaces*  (GUIs), and event-triggered processes that assist our traders, engineers, and operations personnel. This outer layer of our codebase is supported by an inner layer of shared business logic. Business logic is often more complicated than it appears: even a simple question like “what is the next business day for NASDAQ?” involves querying a database of market calendars (a database that requires regular maintenance). So, by centralizing this business logic into a single source of truth, we ensure that all the different systems in our codebase behave coherently.

* To learn about our GUI design process, see: Optimizing UX/UI Design for Trading at HRT.

Even a small change to shared business logic can affect many systems, and we need to check that these systems won’t have issues with our change. It’s inefficient and error-prone for a human to manually verify that nothing is broken. Python’s type annotations significantly improve how quickly we can update and verify changes to shared business logic.

Type annotations allow you to describe the type of data handled by your code. “Type checkers” are tools that reconcile your descriptions against how the code is actually being used. When we update shared business logic, we update the type annotations and use a type checker to identify any downstream systems that are affected.

We also thoroughly document and test our codebase. But written documentation is not automatically synchronized with the underlying code, so maintaining documentation requires a high level of vigilance and is subject to human error. Additionally, automated testing is limited to the scenarios that we test for,* which means that novel uses of our shared business logic will be unverified until we add new tests.

* Generative testing via Hypothesis is a promising new avenue that we are also exploring to address some of the deficiencies of more traditional automated testing.

The rest of this article will explore:

• How type annotations work in Python
• The type checking tools that HRT uses
• An example of detecting downstream issues
• Some tips and tricks for using type annotations effectively

How type annotations work in Python

Let’s look at an example of type annotations to see how they can be used to describe the shape of data. Here’s some type annotated Python code that computes the checksum digit of a CUSIP:

def cusip_checksum(cusip8: str) -> int:
    assert len(cusip8) == 8
    chars: str = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ*@#"
    charmap: dict[str, int] = {
        char: value
        for value, char in enumerate(chars, start=0)
    }
    total: int  = 0
    for idx, char in enumerate(cusip8, start=0):
        value: int = charmap[char]
        if (idx % 2) == 1:
            value *= 2
        total += (value // 10) + (value % 10)
    return (10 - total % 10) % 10

Here’s what the type annotations tell us:

• cusip_checksum() is a function that takes a string as input and returns an integer as output.
• chars is a string.
• charmap is a dictionary with string keys and integer values.
• total and value are integers.

You might notice that the type annotations don’t describe everything about the data. For example:

• The input to cusip_checksum() should be an 8-character string.
• The output of cusip_checksum() should be an integer in the range 0 to 9.
• charmap is a dictionary whose keys are single-character strings.

Even though Python doesn’t have support for refinement types (yet), you can address this in several ways. First of all, this may not be an issue for you: the level of granularity provided just by str and int may be sufficient for your code. Additionally, Python’s typing module provides tools for more advanced type constraints, like NewType(), Literal[], and TypeGuard[].

Type annotations were added in Python 3.5 via a Python Enhancement Proposal (PEP):  PEP 484 – Type Hints. Since then, there have been a number of PEPs to improve type annotations further:

The type checking tools that HRT uses

HRT uses mypy to analyze our Python type annotations. Mypy works by analyzing the type annotations in one or more Python files and determining if there are any issues or inconsistencies.

If something isn’t type annotated, mypy will try to infer the appropriate type. If you want to debug this, you can use the special function reveal_type()(only available in mypy).

Most of the time, mypy is good at type inference, so it’s better to focus on annotating the parameters and return values of a function rather than the internal variables used in a function.

If your codebase doesn’t already have a type checker, it’s also worth evaluating mypy’s competitors: pytype(Google), pyright (Microsoft), and pyre (Meta). And for adding type annotations to existing Python code, take a look at MonkeyType and autotyping.

An example of detecting downstream issues

Here’s a new function, validate_cusip(), that relies on the cusip_checksum() function from earlier:

def cusip_checksum(cusip8: str) -> int:
    ...

def validate_cusip(cusip: str) -> str | None:
    checksum: int
    if len(cusip) == 9:
        checksum = cusip_checksum(cusip[:8])
        if str(checksum) == cusip[8]:
            return cusip
        else:
            return None
    elif len(cusip) == 8:
        checksum = cusip_checksum(cusip)
        return f"{cusip}{checksum}"
    else:
        return None

Mypy is happy with this code:

Success: no issues found in 1 source file

Now, let’s say that we decide we should update cusip_checksum() to return None if it detects that the CUSIP is not valid:

def cusip_checksum(cusip8: str) -> int | None:
    if len(cusip8) != 8:
        return None
    chars: str = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ*@#"
    charmap: dict[str, int] = {
        char: value
        for value, char in enumerate(chars, start=0)
    }
    total: int  = 0
    for idx, char in enumerate(cusip8, start=0):
        try:
            value: int = charmap[char]
        except KeyError:
            return None
        if (idx % 2) == 1:
            value *= 2
        total += (value // 10) + (value % 10)
    return (10 - total % 10) % 10

Mypy automatically detects issues in how validate_cusip() is using cusip_checksum():

error: Incompatible types in assignment (expression has type "int | None", variable has type "int")  [assignment]

Now that we’ve been alerted, we can update validate_cusip() to handle these changes:

def cusip_checksum(cusip8: str) -> int | None:
    ...

def validate_cusip(cusip: str) -> str | None:
    if len(cusip) == 9:
        match cusip_checksum(cusip[:8]):
            case int(checksum) if str(checksum) == cusip[8]:
                return cusip
    elif len(cusip) == 8:
        match cusip_checksum(cusip):
            case int(checksum):
                return f"{cusip}{checksum}"
    return None

In this example, the functions were next to each other in the source code. But mypy really shines when the functions are spread across many files in the codebase.

Some tips and tricks for using type annotations effectively 

Here are some specifics tips and tricks that I’ve learned through experience:

1. Understand the layers of your code. Focus on using type annotations in the inner layers before you focus on type annotations in the outer layers.


2. Understand the limits of type annotations and when you need to add runtime validation. For example, when you are processing data from an external API, typing.TypedDict may be useful for type annotations, but it does not guarantee that the external API will actually provide that data – you will need to verify that the data matches your expected schema at runtime.


3. Be careful about using primitive types (e.g. str, int) to represent domain ideas (e.g. securities, prices). Primitive types, like str/int/float, are easy to understand, but often lack the constraints needed for domain ideas. For example, you could represent a CUSIP like my_cusip: str. Even though a CUSIP can be represented as a string, not all strings are CUSIPs. A CUSIP is 9 characters, using an alphanumeric character set, and the 9th character is a checksum. For this example, you could increase the robustness of your code with a CUSIP type annotations, either by:
   • Creating a CUSIP dataclass (or attrs class)that takes a string and validates that the string is a valid CUSIP upon creation.
   • Using typing.NewType to denote that a CUSIP is a special type of string. For example, CusipStr = NewType(“CusipStr”, str), and a typing.TypeGuard function like is_cusip(text: str) -> TypeGuard[CusipStr].


4. typing.NewType and typing.Literal are lightweight tools for increasing type-safety.In (3.b), CusipStr is lightweight because at runtime the type is still str, without the extra baggage of dataclass/attrs classes. And Literal can act as a lightweight enum.Enum. For example,
   • order_status: Literal["open", "cancelled", "filled"]
   • order_status = "open".
     • Mypy says: Success: no issues found in 1 source file.
   • order_status = "canceled". (Note the typo.)
     • Mypy says:error: Incompatible types in assignment (expression has type "Literal['canceled']", variable has type "Literal['open', 'cancelled', 'filled']") [assignment].

Type annotations have substantial benefits for making your codebase more robust. They are not an all-or-nothing proposition — you can focus on adding type annotations to small parts of your codebase and growing the amount of type annotated code over time. Along with other technologies, Python’s type annotations help HRT to continue thriving in the fast-paced world of global trading.