In the middle of last week, gig economy starlet DoorDash shared a blog post, explaining how they Eliminated Outages. Sounds juicy!
This was an interesting one, less because of its conclusions and more because of its curious tone and dubious content, which hn reliably saw straight-through.
We quickly solved this problem with a simple, Apache Kafka-based asynchronous task processing system that stopped our outages while we continued to iterate on a robust solution.
In 2020, it’s no secret that the Kafka+ZooKeeper stack is a nontrivial investment1 to offer your developers.
This jury’s verdict is that it’s a great stack if you’ve money to burn on a team for maintaining it. You’re safer investing in something boring2, especially “early” on.
Why our legacy solution did not scale
Bafflingly, instead of root-causing their scale problems, they bought a system with 10x the complexity. This didn’t make sense to me, and the decision matrix doesn’t help.
If we don’t understand the failure modes of one (simpler) tool, why should Kafka treat us any better?
This brought back memories of googling for documentation on some kafka feature.
Each query involved sifting through pages of medium articles (!), and a brigade of snake oil vendors.
Weird, but whatever.
We were using the largest available single-node RabbitMQ solution that was available to us. There was no path to scale vertically any further.
One positive takeaway is that a single, naively-operated, and outdated RabbitMQ instance can power a company with $1 billion in revenue. That’s nothing to scoff at!
For those of us who care to seed our innovation tokens in meadows far away from Kafka, ZooKeeper, or “growth-hacks”, this is actually a good advertisement for RabbitMQ.
Read from another perspective, this post kind of says:
RabbitMQ a tool that works well, and has chosen it’s constraints purposefully. If your requirements are a problem for RabbitMQ, the paths around them are well-trodden. RabbitMQ is boring, and that’s a good thing for a tool to be.
The discussion on hn also saw a case study contributed from zbentley of Klaviyo. If this had been a blog post, I would have gladly written about them, instead! 😊
The same day, there was an interesting product launch on HN, of a consumer-review style website.
Apparently, it’s the latest iteration of a not-super-niche trend, of investing in reliable and durable consumer goods3.
This is an interesting break from our supposedly natural, neophilic norms.
A recent pair of NPR productions, however, might frame this as a regression to the mean!
The first documents how oil companies invented recycling, to train the public to throw things away, so that they would purchase more plastic.
As it turns out, of course, recycling was essentially4 a myth from day 1. Checkmate, consumers.
But yeah! People had to be convinced to throw things away5, because the concept of “single-use” anything was quite unusual: “Reduce” and “Reuse” were not things we needed reminding about in the 1940s, it was just how people lived.
Now that recycling is a confirmed myth, and we’re chastised for hoarding (reuse) and overconsumption (reduce), it’s sensible that that consumers would revert to investing in durable and repairable6 goods.
If you’re going to be hoarding that socks and sock-drawer for the next 30 years, they’d better keep your feet warm.
Last week, I linked to an article about all the ways the election could go off the rails, should the results be “contestable”. That article was pretty spooky, especially if you imagined all possible outcomes with equal probabilities.
Of course, these outcomes are not equally likely, but it’s hard for you and me, mere laypeople that we are, to know which to bet on.
This week, there was some interesting news from the NY Times, indicating that the market may have settled on an answer.
Investors are of the view that a “blue wave” victory — in which Democrats retain the House of Representatives and retake the Senate as well as the presidency — represents the best chance to get another large injection of federal money into an economy that continues to struggle.
The hypothesis is that a Democratic Party win would yield a booster shot to the economy, whereas more of the current administration would imply a slower convalescence.
The finance folks have a lot to gain (or lose!) from making good bets on that spread. And, if there’s any group of people that’s good at pricing-in the future, it’s the finance folks7!
To tease out the underlying views of investors, analysts at JPMorgan Chase & Company recently assembled baskets of shares in companies they see as potential winners or likely losers in the event of a Biden victory.
Since early September, “the Democrat Agenda Outperformers have gained 10 percent relative to the Underperformers baskets, suggesting the U.S. equity markets have been pricing in a higher probability of a Biden Presidency,” the JPMorgan analysts wrote in a research note published last week.
“The odds of a Biden victory continue to rise, as do the chances of a blue sweep, and if the recent bid for risk assets is any guide, such an eventuality could prove a decidedly positive event for domestic equities” Ian Lyngen, a bond market analyst with BMO Capital Markets, wrote earlier this week.
This is wall-street speak for “We’re pretty sure Biden is going to win”.
Since the short-term futures of these economic segments hinge on the outcome of the election8, this is one of those rare moments where we have a prediction market on a singular event.
With that, do we even need an election? If the “Outperformers Basket” is up by more than 10% on November 3rd, maybe we can spare ourselves election week drama, and put the Democrats in.
Kafka+ZK are two interacting Apache products, each targeting the JVM. This gives them three compounding complexity points (interaction, Apache, JVM), and implies attack surface, foot-guns, and spinny-knobs galore.↩︎
Boring is used here, in the fashion of “it just works”. Technology is boring if it’s been around the block a while, is inexpensive to operate, and has an engaging community. For more on this topic, check out this slide deck.↩︎
The hackernews discussion has 403 comments and 1376
points! There are also some really good product recommendations in
there. grep http.↩︎
Well, at least 9% is recyclable.↩︎
And why should you throw a good thing away? Think of that one plastic bag full of plastic bags your mom has in her coat closet. Try to imagine a space with more latent potential!↩︎
In local news, it seems like the local right to repair bill is polling positively, in spite of auto manufacturers’ scare campaigns against the proposed legislation. It’s nice to see citizens leaning in the direction of responsible ownership.↩︎
Assuming you buy in to the efficient-market hypothesis. Although, it has been fashionable to non-ironically flout it these days, so take this whole section with a grain of salt.↩︎
That is, unless the Republican party caves in and permits the idea of a stimulus bill. Trump has suggested that he wants such a bill, but the Republican leaders in the senate have preferred to rush a vote on the SCOTUS justice, instead.↩︎