I’m by no means an expert on this topic, but what about using predefined hash map of value to function call/closure unless that also yields branched programming underneath? Additionally, maybe you want to use macros of some kind to generate the “magic” function you’re looking for, where it constructs a single branchless statement up to N terms, whether it’s procedural macros in Rust or C macros.
I find defaultdict, OrderedDict, namedtuple among other data structures/classes in the collections module to be incredibly useful.
Another module that's packaged with the stdlib that's immensely useful is itertools. I especially find takewhile, cycle, and chain to be incredibly useful building blocks for list-related functions. I highly recommend a quick read.
EDIT: functools is also great! Fantastic module for higher-order functions on callable objects.
I mostly migrated to frozen dataclasses from namedtuples when dataclasses became available. I’m curious about your preference for the namedtuple. Is it the lighter weight, the strong immutability, the easy destructing? Or is it that most tuples might as well be namedtuples? Those are the advantages I can think of anyway :)
You could of course accomplish the same with a dictionary comprehension, but I find this to be less noisy. Also, they have `_asdict()` should you want to have the contents as a dict.
You don't need `_make()` with dataclasses, and you get `asdict()` as a stand-alone function so it doesn't clash with each class's namespace. Here's what your code might look like with them:
import sqlite3
from dataclasses import asdict, dataclass
@dataclass
class EmployeeRecord:
name: str
age: int
title: str
department: str
paygrade: str
conn = sqlite3.connect("/companydata")
cursor = conn.cursor()
cursor.execute("SELECT name, age, title, department, paygrade FROM employees")
for emp in (EmployeeRecord(*row) for row in cursor.fetchall()):
print(emp.name, emp.title)
print(asdict(emp))
Dictionary comprehensions can be very elegant. List and dictionary comprehensions are very powerful and expressive abstractions. In fact, while not good practice you can pretty much write all Python code inside comprehensions including stuff regarding mutation.
This is valid(as in it will run, but highly unidiomatic) code:
quicksort = lambda arr: [pivot:=arr[0], left:= [x for x in arr[1:] if x < pivot], right := [x for x in arr[1:] if x >= pivot],
quicksort(left) + [pivot] + quicksort(right)][-1] if len(arr) > 1 else arr
That's incredibly clever, generators are underrated. I once challenged my friend to do leetcode problems with only expressions. Here's levenshtein distance, however it's incredibly clunky.
levenshtein_distance = lambda s1, s2: [matrix := [[0] * (len(s2) + 1) for _ in
range(len(s1) + 1)], [
[
(matrix[i].__setitem__(j, min(matrix[i-1][j] + 1, matrix[i][j-1] +
1, matrix[i-1][j-1] + (0 if s1[i-1] == s2[j-1] else 1))), matrix[i][-1])[1]
for j in range(1, len(s2) + 1)
]
for i in range(1, len(s1) + 1)
], matrix[-1][-1]][-1]
By default, List[Tuple]]. You could do a list comp over the fetchall(), but at that point there’s already some magic happening, so why not make it explicit?
If I find my self write a[0] a[1] a[2] in more than one place, I would upgrade it to a namedtuple. Much better readability, can be defined inline like `MyTuple = namedtuple('MyTuple', 'k1 k2 k3')`
For anyone wanting some more explanation, ChainMap can be used to build nested namespaces from a series of dicts without having to explicitly merge the names in each level. Updates to the whole ChainMap go into the top-level dict.
The docs are here [0].
Some simple motivating applications:
- Look up names in Python locals before globals before built-in functions: `pylookup = ChainMap(locals(), globals(), vars(builtins))`
- Get config variables from various sources in priority order: `var_map = ChainMap(command_line_args, os.environ, defaults)`
I found it perfect for structured logging, where you might want to modify some details of the logged structures (e.g. a password) without changing the underlying data.
I just wish python had some better ergonomics/syntactic sugar working with itertools and friends. Grouping and mapping and filtering and stuff quickly become so unwieldy without proper lambdas etc, especially as the typing is quite bad so after a few steps you're not sure what you even have.
Just as recent as today I went to Kotlin to process something semicomplex even though we're a python shop, just because I wanted to bash my head in after a few attempts in python. A DS could probably solve it minutes with pandas or something, but again stringly typed and lots of guesswork.
(It was actually a friendly algorithmic competition at work, I won, and even found a bug in the organizer's code that went undetected exactly because of this)
I find converting things from map objects or filter objects back to lists to be a bit clunky. Not to mention chaining operations makes it even more clunky. Some syntatic sugar would go a long way.
Two dictionaries with equal keys and values are considered equal in Python, even if the order of the entries differ. By contrast, two OrderedDict objects are only equal if their respective entries are equal and if their order does not differ.
It may be more explicit: OrderedDict has move_to_end() which may be useful e.g., for implementing lru_cache-like functionality (like deque.rotate but with arbitrary keys).
OTOH that’s a lot less useful now that functools.lru_cache exists: it’s more specialised so it’s lighter, more efficient, and thread-safe. So unless you have extended flexibility requirements around your LRU, OD loses a lot there.
And if you’re using a FIFO cache, threading a regular dict through a separate fifo (whether linked list or deque) is more efficient in my experience of implementing both S3 and Sieve.
Also, dicts can become unordered at any time in the future. Right now the OrderedDict implementation is a thin layer over dict, but there are no guarantees it’ll always be that.
dict are ordered to keep argument order when using named arguments in function calling. So it would be a non-trivial breaking change to revert this now.
I would argue that OrderedDict have more chances to be depreciated than dict becoming unordered again, since there is now little value to keep OrderedDict around now (and the methods currently specific to UnorderedDict could be added to dict).
I guess for most purposes, OrderedDicts are then obsolete, but I believe there are some extra convenience methods that they have, but I've only really needed to preserve order.
Makes you think what other parts of Python have become obsolete.
It worked as an accidental implementation detail in CPython from some other optimization, but it wasn't intentional at the time. Because it wasn't intentional and wasn't part of the spec, that code could be incompatible with other interpreters like pypy or jython.
The reason Guido didn't want 3.6 to guarantee dict ordering was to protect 3.5 projects from mysteriously failing when using code that implicitly relied on 3.6 behaviors (for example, cutting and pasting a snippet from StackOverflow).
He thought that one cycle of "no ordering assumptions" would give a smoother transition. All 3.6 implementations would have dict ordering, but it was safer to not have people rely on it right away.
As someone who just had to backport a fairly large script to support 3.6, I found myself surprised at how much had changed. Dataclasses? Nope. `__future__.annotations`? Nope. `namedtuple.defaults`? Nope.
It's also been frustrating with the lack of tooling support. I mean, I get it – it's hideously EOL'd – but I can't use Poetry, uv, pytest... at least it still has type hints.
I use defaultdict a lot - for accumulators when you're not sure about what is coming. Here's a simplified example:
# a[star_name][instrument] = set of (seed, planet index) of visited planets
a = defaultdict(lambda: defaultdict(set))
for row in rows:
a[row.star][row.inst].add((row.seed, row.planet))
This is a dict-of-dict-of-set that is accumulating from a stream of rows, and I don't know what stars and instruments will be present.
Definitely very excited to see this be a thing. Genuinely liked the approach to make function calls strongly typed and rely on functional programming principles.
During my senior year, I worked on a research project very similar to this and I’m glad to see this out there for everyone. I’d love to connect with the team if possible!
I’m always amazed by the level of the discussions on Signals and Threads. I especially enjoyed hearing about the challenges with performance engineering in OCaml versus C++. The work on unboxed types is very promising.
This is unironically my plan/playbook with ideas I have for startups. Start with Python, node, or whatever. Then go to Rust once product-market fit has been established.
Rust really hits that sweet spot with memory-management, programming paradigm flexibility, and speed.
It’s precisely great for what I need because it’s essentially an ML with good imperative constructs.
I loved reading the “making of” post. I’m very interested in getting into technical writing and your post, really helped me understand your thought process behind “Elixir at Ramp”.
Totally agree, when is everything is functions and data, it makes life a lot easier. I especially find it useful when refactoring code while trying to keep an existing function signature.
Having spent considerable time in both Java and Elixir, I would also choose Kotlin (or Java) over Elixir as a backend language.
I worked in Elixir for over a year, and frankly was quite disappointed with it. Elixir has a lot of good ideas, but falls short in some crucial points in execution. Relying on shapes instead of types to identify results / objects, weird syntax choices (You use both <%= %> and {} to interpolate in phoenix templates, depending on where you are), no ability to write custom guards, a lot of language concept overlap - for example behaviors / protocols should be one implementation instead of two separate ideas.. Elixir is an okay language, but I think it's just a fad, not good enough to have staying power. I think a better language written on the BEAM will come along and be the language that Elixir should have been. Just my personal opinion.
> Relying on shapes instead of types to identify results / objects
Is the issue lack of types or relying on shapes? Those can be orthogonal features, since you can have structural typing (i.e. types that are built on top of shapes). Nominal typing is often too rigid (and not the direction we are exploring in Elixir's type system).
> for example behaviors / protocols should be one implementation instead of two separate ideas
Elixir has three building blocks: processes, modules, and data. Message-passing is polymorphism for processes, behaviour is polymorphism for modules, and protocol is polymorphism for data.
The overlapping comes that modules in Elixir are first-class (so they can be used as data) and that data (structs) must be defined inside modules. So while maybe those features could feel closer to each other, I don't think they could be one implementation. One idea would be to treat modules/behaviors as "nullary protocols" but that's not what "nullary typeclasses" mean in Haskell, so it is quite debatable.
Do you have examples of languages that do both in one implementation? For example, my understanding is that both Haskell and Go offer "protocols" but not "behaviours". I'm not aware of a language with both as one.
I'd love to hear other concepts that overlap, it is always good food for thought!
Re: "You use both <%= %> and {} to interpolate in phoenix templates, depending on where you are" - no need for EEx anymore for web development, just use HEEx, which has standardized on {}. You can use HEEx for LiveViews and "DeadViews" (server-side rendered Phoenix)
I don't recall 100%, but I think this is a BEAM feature that exists because they don't want to run arbitrary code as part of guards that could have side effects, delays and so on. I don't remember the specifics.
Interesting. We had the exact opposite reaction. We switched from Java to Elixir at my current company and we'll never go back. It has been a two-year migration, but worth every second. We're twice as productive in Elixir, it's easier to test and deploy, and we don't have to paint IntelliJ lipstick on the language to hide all the shitty parts.
Doordash uses Kotlin for backend as well. I suppose they want FE/BE to be in the same language. And when your main product is a native app Kotlin starts to make sense.
