Hudson River Trading’s (HRT’s) Python codebase is large* and constantly evolving. Millions of lines of Python reflect the work of hundreds of developers over the last decade. We trade in over 200 markets worldwide — including nearly all of the world’s electronic markets — so we need to regularly update our code to handle changing rules and regulations.
* To learn how we handle large-scale code organization, see: Building a Dependency Graph of Our Python Codebase.
Our codebase provides command-line interface (CLI) tools, graphical user interfaces* (GUIs), and event-triggered processes that assist our traders, engineers, and operations personnel. This outer layer of our codebase is supported by an inner layer of shared business logic. Business logic is often more complicated than it appears: even a simple question like “what is the next business day for NASDAQ?” involves querying a database of market calendars (a database that requires regular maintenance). So, by centralizing this business logic into a single source of truth, we ensure that all the different systems in our codebase behave coherently.
* To learn about our GUI design process, see: Optimizing UX/UI Design for Trading at HRT.
Even a small change to shared business logic can affect many systems, and we need to check that these systems won’t have issues with our change. It’s inefficient and error-prone for a human to manually verify that nothing is broken. Python’s type annotations significantly improve how quickly we can update and verify changes to shared business logic.
Type annotations allow you to describe the type of data handled by your code. “Type checkers” are tools that reconcile your descriptions against how the code is actually being used. When we update shared business logic, we update the type annotations and use a type checker to identify any downstream systems that are affected.
We also thoroughly document and test our codebase. But written documentation is not automatically synchronized with the underlying code, so maintaining documentation requires a high level of vigilance and is subject to human error. Additionally, automated testing is limited to the scenarios that we test for,* which means that novel uses of our shared business logic will be unverified until we add new tests.
* Generative testing via Hypothesis is a promising new avenue that we are also exploring to address some of the deficiencies of more traditional automated testing.
The rest of this article will explore:
- How type annotations work in Python
- The type checking tools that HRT uses
- An example of detecting downstream issues
- Some tips and tricks for using type annotations effectively
How type annotations work in Python
Let’s look at an example of type annotations to see how they can be used to describe the shape of data. Here’s some type annotated Python code that computes the checksum digit of a CUSIP:
def cusip_checksum(cusip8: str) -> int:
assert len(cusip8) == 8
chars: str = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ*@#"
charmap: dict[str, int] = {
char: value
for value, char in enumerate(chars, start=0)
}
total: int = 0
for idx, char in enumerate(cusip8, start=0):
value: int = charmap[char]
if (idx % 2) == 1:
value *= 2
total += (value // 10) + (value % 10)
return (10 - total % 10) % 10
Here’s what the type annotations tell us:
cusip_checksum()
is a function that takes a string as input and returns an integer as output.chars
is a string.charmap
is a dictionary with string keys and integer values.total
andvalue
are integers.
You might notice that the type annotations don’t describe everything about the data. For example:
- The input to
cusip_checksum()
should be an 8-character string. - The output of
cusip_checksum()
should be an integer in the range 0 to 9. charmap
is a dictionary whose keys are single-character strings.
Even though Python doesn’t have support for refinement types (yet), you can address this in several ways. First of all, this may not be an issue for you: the level of granularity provided just by str
and int
may be sufficient for your code. Additionally, Python’s typing module provides tools for more advanced type constraints, like NewType()
, Literal[]
, and TypeGuard[]
.
Type annotations were added in Python 3.5 via a Python Enhancement Proposal (PEP): PEP 484 – Type Hints. Since then, there have been a number of PEPs to improve type annotations further:
Type Annotation PEPs
- PEP 482 – Literature Overview for Type Hints
- PEP 483 – The Theory of Type Hints
- PEP 484 – Type Hints
- PEP 544 – Protocols: Structural subtyping (static duck typing)
- PEP 560 – Core support for typing module and generic types
- PEP 563 – Postponed Evaluation of Annotations
- PEP 585 – Type Hinting Generics In Standard Collections
- PEP 586 – Literal Types
- PEP 589 – TypedDict: Type Hints for Dictionaries with a Fixed Set of Keys
- PEP 591 – Adding a final qualifier to typing
- PEP 593 – Flexible function and variable annotations
- PEP 604 – Allow writing union types as X | Y
- PEP 612 – Parameter Specification Variables
- PEP 613 – Explicit Type Aliases
- PEP 646 – Variadic Generics
- PEP 647 – User-Defined Type Guards
- PEP 649 – Deferred Evaluation Of Annotations Using Descriptors
- PEP 655 – Marking individual TypedDict items as required or potentially-missing
- PEP 673 – Self Type
- PEP 675 – Arbitrary Literal String Type
- PEP 692 – Using TypedDict for more precise **kwargs typing
- PEP 695 – Type Parameter Syntax
- PEP 696 – Type defaults for TypeVarLikes
- PEP 698 – Override Decorator for Static Typing
- PEP 702 – Marking deprecations using the type system
- PEP 705 – TypedMapping: Type Hints for Mappings with a Fixed Set of Keys
The type checking tools that HRT uses
HRT uses mypy to analyze our Python type annotations. Mypy works by analyzing the type annotations in one or more Python files and determining if there are any issues or inconsistencies.
Given this Python… | Mypy will say… |
x = 3 | Success: no issues found in 1 source file |
x: int = 3 | Success: no issues found in 1 source file |
x: str = 3 | error: Incompatible types in assignment (expression has type “int”, variable has type “str”) [assignment] Found 1 error in 1 file (checked 1 source file) |
If something isn’t type annotated, mypy will try to infer the appropriate type. If you want to debug this, you can use the special function reveal_type()
(only available in mypy).
Given this Python… | Mypy will say… |
x = 3 | note: Revealed type is “builtins.int” Success: no issues found in 1 source file |
hex_digits = {
| note: Revealed type is “builtins.dict[builtins.str, builtins.int]”Success: no issues found in 1 source file |
Most of the time, mypy is good at type inference, so it’s better to focus on annotating the parameters and return values of a function rather than the internal variables used in a function.
If your codebase doesn’t already have a type checker, it’s also worth evaluating mypy’s competitors: pytype(Google), pyright (Microsoft), and pyre (Meta). And for adding type annotations to existing Python code, take a look at MonkeyType and autotyping.
An example of detecting downstream issues
Here’s a new function, validate_cusip()
, that relies on the cusip_checksum()
function from earlier:
def cusip_checksum(cusip8: str) -> int:
...
def validate_cusip(cusip: str) -> str | None:
checksum: int
if len(cusip) == 9:
checksum = cusip_checksum(cusip[:8])
if str(checksum) == cusip[8]:
return cusip
else:
return None
elif len(cusip) == 8:
checksum = cusip_checksum(cusip)
return f"{cusip}{checksum}"
else:
return None
Mypy is happy with this code:
Success: no issues found in 1 source file
Now, let’s say that we decide we should update cusip_checksum()
to return None
if it detects that the CUSIP is not valid:
def cusip_checksum(cusip8: str) -> int | None:
if len(cusip8) != 8:
return None
chars: str = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ*@#"
charmap: dict[str, int] = {
char: value
for value, char in enumerate(chars, start=0)
}
total: int = 0
for idx, char in enumerate(cusip8, start=0):
try:
value: int = charmap[char]
except KeyError:
return None
if (idx % 2) == 1:
value *= 2
total += (value // 10) + (value % 10)
return (10 - total % 10) % 10
Mypy automatically detects issues in how validate_cusip()
is using cusip_checksum()
:
error: Incompatible types in assignment (expression has type "int | None", variable has type "int") [assignment]
Now that we’ve been alerted, we can update validate_cusip()
to handle these changes:
def cusip_checksum(cusip8: str) -> int | None:
...
def validate_cusip(cusip: str) -> str | None:
if len(cusip) == 9:
match cusip_checksum(cusip[:8]):
case int(checksum) if str(checksum) == cusip[8]:
return cusip
elif len(cusip) == 8:
match cusip_checksum(cusip):
case int(checksum):
return f"{cusip}{checksum}"
return None
In this example, the functions were next to each other in the source code. But mypy really shines when the functions are spread across many files in the codebase.
Some tips and tricks for using type annotations effectively
Here are some specifics tips and tricks that I’ve learned through experience:
- Understand the layers of your code. Focus on using type annotations in the inner layers before you focus on type annotations in the outer layers.
-
Understand the limits of type annotations and when you need to add runtime validation. For example, when you are processing data from an external API,
typing.TypedDict
may be useful for type annotations, but it does not guarantee that the external API will actually provide that data – you will need to verify that the data matches your expected schema at runtime. -
Be careful about using primitive types (e.g. str, int) to represent domain ideas (e.g. securities, prices). Primitive types, like str/int/float, are easy to understand, but often lack the constraints needed for domain ideas. For example, you could represent a CUSIP like
my_cusip: str.
Even though a CUSIP can be represented as a string, not all strings are CUSIPs. A CUSIP is 9 characters, using an alphanumeric character set, and the 9th character is a checksum. For this example, you could increase the robustness of your code with a CUSIP type annotations, either by:- Creating a CUSIP dataclass (or attrs class)that takes a string and validates that the string is a valid CUSIP upon creation.
-
Using typing.NewType to denote that a CUSIP is a special type of string. For example,
CusipStr = NewType(“CusipStr”, str)
, and atyping.TypeGuard
function likeis_cusip(text: str) -> TypeGuard[CusipStr]
.
-
typing.NewType and typing.Literal are lightweight tools for increasing type-safety.In (3.b),
CusipStr
is lightweight because at runtime the type is still str, without the extra baggage ofdataclass/attrs
classes. AndLiteral
can act as a lightweight enum.Enum. For example,-
order_status: Literal["open", "cancelled", "filled"]
-
order_status = "open".
- Mypy says: Success: no issues found in 1 source file.
-
order_status = "canceled"
. (Note the typo.)- Mypy says:
error: Incompatible types in assignment (expression has type "Literal['canceled']", variable has type "Literal['open', 'cancelled', 'filled']") [assignment].
- Mypy says:
-
Type annotations have substantial benefits for making your codebase more robust. They are not an all-or-nothing proposition — you can focus on adding type annotations to small parts of your codebase and growing the amount of type annotated code over time. Along with other technologies, Python’s type annotations help HRT to continue thriving in the fast-paced world of global trading.