Should dependencies always be kept up to date?

I work in the Developer Experience (DevX) realm right now, and the current perspective on handling the potential security concerns that Mythos is highlighting is that we should immediately update all code (third party) dependencies to their respective, latest versions.

While that perspective is valid for my company right now (there’s so much more to the story there), this isn’t always a valid perspective to take.

And this isn’t a new concept, either.

If you go deep underground into the most secure places like government cold storage or nuclear facilities, you’ll often find ancient technology. Think 8 inch floppy disks still running core infrastructure from post-WW2 days.

Why would you let something languish like this?

Well, updating a nuclear reactor to be fully online suddenly potentially exposes its edges to the world. And getting security protocols to work effectively is hard. And scary when you consider the real life damage that nuclear material can cause if handled incorrectly when a bug is triggered.

The Unix / Linux ecosystem has pockets of this as well - a lot of core utils like ls and cat are generally full featured for what they were intended and don’t exactly need to be updated. Because, well, it does what it does, and people have built other critical infra on top of this expected behavior.

I digress a bit, but my bigger point in today’s society is a bit more like this:

If you are concerned about your old dependencies leaving you exposed to zero-day bugs and exploits, you should also be equally as concerned about new ones introducing you to new ones.

Sure, it’s highly likely that newer versions of code are far more secure than the old ones, but the chance that the newer versions introduced (or reintroduced) past bugs is certainly not zero.

I’ve worked with a lot of engineers over the years who immediately shut down attempts to update their project dependencies. It’s introducing unknowns into code that “works just fine” in most cases, at least when it comes to competent and often old-school engineers.

On one hand, I really don’t like that mentality. But on the other, I kind of revere it. We are constantly asking ourselves in software questions like “but when is it done?” and rarely have an answer.

So make sure your dependency graph is as up to date as is reasonable. It’s a good practice to have!

But also don’t do it and expect yourself to be immune to future late-night bug-bashes once Mythos is released to the world.

Basically this model is quite advanced to the point where it is discovering bugs and “zero day” (aka unknown) issues on 30 year old tooling that no one has touched for years.

Which is alarming.

I suppose.

At least I know my company is currently freaking out about it. As is nearly every other company inundated with massive amounts of tech debt who have willfully turned a blind eye to it for literal years. Suddenly we all have only six weeks to get ourselves prepared for this (dystopian?) future.

Present freaking-out excluded, we’ve also managed to do a whole lot in the world of computers over the last 30 years. No one was the wiser as to these long-dormant bugs.

Say Anthropic never created this model, either intentionally or not. Would that actually be a problem in our world today?

Our AI overloads would definitely use the same excuse “if not us then someone else!” so we should feel grateful that they’re telling us all ahead of its release.

It honestly sounds a lot like the ever-looming quantum future of computing.

Quantum computing, the little if anything I know about it, is essentially computing that does not rely on typical zeros and ones when it comes to representing data in memory registers. Imagine if you had a computer that could handle anything from zero to, say, a hundred within each bit in some way.

Today’s cryptography just wouldn’t keep up in that world. So things like 2048 bit encryption really have no chance against a potential threat like that. At least as I understand it.

Some of this feels similarly.

Progress for the sake of progress isn’t actually progress for humankind.

Not always at least.

My opinion here is more political actually. If we never allowed a company to quite literally steal the world’s data to train their models, then perhaps they’d never have created Skynet, I mean, Mythos in the first place.

Would we be worse off for it? Probably not.

Meanwhile, in the trenches of my day to day job…

I had a pull request to review today that had over 351 comments on it, 35 different commits, modifying only 3 files, nearly all of which was generated by Claude. A huge portion of it was rewriting slight variations of other code that already existed in the same repository. Or better yet, redefining standard library code, a known issue in a lot of these actions cited by others.

This is proposed core build CI infrastructure code!

So on one side, we have these ridiculously overpowered and world-ending models put out by companies worth trillions of dollars…

And on the other hand, we have a bunch of AI slop becoming ever-increasingly relied on by lazy engineers who are squeezed to death by management who really have no idea how else to consider themselves successful in their jobs.

What a world we live in.

I went earlier on a walk with my dog and saw people out on their evening walks. They were on their bikes riding home. People were active, enjoying the beautiful weather outside today.

If we accidentally (or purposely, slowly) sign on to having AI run our bank accounts and suddenly we onboard Skynet to the world around us, what’s the point of it all?

We’re human, we should be doing human things. Isn’t that the point of technology, to better add to our lives?

If instead of usurp the beauty that is life with AI models that start to make autonomous decisions and we get lazier and lazier in letting it do all the important work for us, we rob ourselves of a better life. We instead place all control and money into the hands of a few who won’t know how to utilize it when that day comes.

Or worse yet, they’ll simply weaponize it against us. Imagine rioters with pitchforks charging the nearest data center, but then seeing others fight against the action because, well, that’s their banking and picture data too!

I imagine it to be somewhere between Wall-E and the Terminator movies.

Realistically, this is where government oversight and anti-trust laws should help us balance new features and technology with the desire to be human.

So again…

Just because we can doesn’t mean we should.

Regardless, the primary problem coming for AI is very similar to what caused some issues during the early days of the internet. Back then, the internet was new and companies were scrambling to build a bunch of high throughput fiber optic cable across the US.

Unfortunately, the “last mile” problem still existed, and despite you maybe living only a mile away from one of the fiber hubs, you still had a 50 year old phone line trying to send that fiber optic internet to your house.

womp, womp

This created a weird problem during the early days of high speed internet - when, initially, the internet backbones had effectively too much capacity compared to their actual use. “Dark” fiber lines actually existed at this time, meaning that they were either not transmitting internet traffic due to lack of the need for it, or they were simply under utilized.

Imagine having spent a bunch of money to build those lines, only to realize that they weren’t being used as you had hoped. How did that look on your balance sheets?

Fast forward to today.

Add to our story an oblivious billionarie, Kevin Leary, of Shark Tank notoriety. In particular, let’s talk about his actions this past week where he successfully strongarmed a set of local county board members in Utah to approve his “Stratos” data center.

This is a 40,000 acre data center that’s going to use up to 9 gigawatts of power.

That’s a data center that’s 8 miles long. It’s bigger than the entirety of Bryce Canyon national park.

And 9 gigawatts of power is 3 times as much power as the entire state of Utah currently consumes!

Besides this effort being terribly mishandled optically, it also showcases a huge problem that Wall Street has yet to come to grips with.

The demands for AI model training and serving is simply insatiable and do not align with real world constraints.

And this isn’t because I’m getting grumpier with time and hate progress in the tech space. Part of this is due to the sheer amount of compute that’s required to train new models these days, a number that seems to only increase over time.

Let’s explain.

Imagine with me that this thing is built and it starts consuming 9 gigawatts of power, and let’s ignore other issues like pollution and its water usage in the desert region that is Utah.

Given that Utah taxpayers are already vying for their own usage of 3 gigawatts of power (in the form of their electric bills), a few things will necessarily have to happen:

• more power plants will have to be built, along with more infrastructure to transmit more power to more places.
• prices for electricity with have to go up to effectively throttle what is available amongst all interested parties.

In the first case, you can’t exactly stand up a bunch of new power plants and infrastructure overnight. You need to truck in concrete, do environmental reviews, hire engineers, design plans, and a lot more - all of which take time and (currently) require a lot of approvals. Are these power plants going to be run via coal or gas? Who has to live next to them?

What if Utah simply buys electricity from a neighboring state?

Same story is going to apply here. Electricity requires infrastructure, especially over long distances. Given normal economic supply and demand, neighboring states may find their own electricity infrastructure strained to supply Utah’s needs, which typically means their prices will go up too.

All because I want to occasionally generate a picture of a banana or create a fun borderline-copyright-infringed avatar for my profile somewhere.

I digress.

In the second case from above, you’re effectively taunting the public with class warfare. Basically, you’re enacting Hunger Games rules.

There’s already a growing anti-billionaire sentiment in this country, and giving special tax deals to them while footing Joe the taxpayer with the bill is not going to be an easy sell. Even in the conservative state of Utah.

There’s a third thing that may happen here.

This is why I mentioned the fiber-optic story earlier.

What if the data center is built but Utah can’t supply it with enough power?

Does this mean that the data center is going to sit half used or unused entirely? What kind of environmental impact is that going to additionally add here?

What will happen to all the money that various investors put into this project? Are they going to disappear to zero via a plummeting stock price? Is there another taxpayer funded bailout in the works?

The takeaway?

Do we really want our public infrastructure, paid for with public taxpayer funds, to go to sweetheart deals with billionaires and/or multinational companies? The same ones will like to “socialize the costs, privatize the profits”?

If the past is any indication of the future, all the dollars they receive from the actions made via these data centers aren’t exactly going to be shared by the public who helped facilitate the building of them.

Not to mention that companies like these will abandon these locations when they are no longer profitable to maintain them.

So I guess the real question comes down to this - with the limited resources of our existing power grid, do we feel like the best usage of it should be to power the data centers at the heart of a few tech companies pumping out new AI models?

The same models that are becoming increasingly expensive, by the way, per subscription. Let’s not forget that!

I think you already know my opinion on the matter.

You can be as loyal as you want to a company, they will never be loyal back.

I think it goes without saying in today’s environment, but I think it’s important to reiterate again.

Today, Oracle laid off 30,000 employees, many of whom have been there 30 or more years!

It’s the biggest tech layoff in the last 12 months, and it’s absolutely sad to see what appears to be an arbitrary algorithm that selected the employees to be cut.

What makes this particularly galling, too, is that last year, Oracle posted a 95% increase in net income!

An additional claimed reason for this layoff is that this action will free up more cash for data center build out.

For AI, of course.

This LinkedIn blog post stated some of the above, but chose to take the route of comparing Oracle with IKEA, a business that they feel is weathering the current storm very differently.

This all gives me pause.

I’ve been where I’ve been for a long time, at least a long time for “big tech”. I’ve seen a few layoffs already and while I’ve been lucky to not get caught up in any of them, I feel like just because I’m here does not mean I need to stay out of a sake of “loyalty”.

Companies will cut you without a second thought, whether you’ve been there 3 years or 33 years, so do not expect anything less from them.

That is all for today.

The EU has stronger labor laws.

This manifests itself in a few ways that maybe you wouldn’t consider.

When interviewing for a European-based company, and one that isn’t necessarily an offshoot of an existing company based out of the US, job interviews tend to be more formal and a bit more tech-direct skills (i.e., not soft skills) focused.

Reddit has a lot more to read on the topic.

This showcases in how people are hired and fired.

From bluematterconsulting.com, I found a good description of this:

Across Europe, the concept of at-will employment does not exist, so employees have much greater protections than in the US. If an employer wants to terminate an employee, then it must do so for well-documented, established reasons. In addition, the process of dismissing an employee is usually a lengthy one, with required warnings, consultations, notice periods, and the like. An employer that fails to follow the proper procedures and does not afford an employee due process is open to a wrongful termination suit.

Hiring definitely feels more deliberate over there, and when layoffs occur, companies can’t simply do it overnight and uproot people’s lives like they do here in the US (ahem, healthcare).

I’ve known managers who have had to handle a layoff for some of their EU based employees and doing so took them months. They also had to show legal proof as to why they were no longer needed.

Feedback after interviewing is also different.

When we get rejected from a potential future employer during the course of an interview cycle, we here in the US are largely told “sorry, but no thank you” via a simple email.

You really don’t get much more than that.

You basically forever get to ask yourself “What did I do wrong?” or “Why did they not hire me?” and you basically will never know the true answer unless you get very lucky with the recruiter you worked with and they give you a little bit more detail.

It’s not typical for them to provide anything specific to you due to the market (and law) favoring the employer side of the relationship.

However, when interviewing for an EU based position, you’ll absolutely get told why you weren’t hired. It’s probably not very easy to hear, but it does afford you the opportunity to uplevel yourself for the next time you apply elsewhere.

Personally, I’d rather have the EU based approach.

Too many times I’ve been told “sorry, but no” leaving only weird, unanswered questions in my head and my psyche.

Jensen Huang is the current CEO for Nvidia, the man being credited with taking the company to stratopheric heights in the last 10 years. This is mainly because Nvidia creates the best GPUs that are used in the core model training algorithms for anything AI related.

What I wanted to quickly write about was about something he said during a recent interview with Jim Cramer (yes, that Jim Cramer) that reasonated with me.

We currently live in a world where a huge portion of the tech community, i.e., software engineers particularly, live in a near constant state of anxiety. Their (our) jobs are potentially being automated away and it seems that we can’t go a single quarter without hearing about some huge company laying off a bunch of employees.

The justification? AI.

Never mind the old adage “putting the cart before the horse”, firing employees before you’ve actually realized the net gain from the technology that claims to reduce the need for them.

Apparently Jenson sings a slightly different tune. First, the setup:

Huang was talking about the ability of AI to elevate the capabilities of humans when CNBC’s Jim Cramer asked him why companies are laying off people, saying they want to do more with less.

His response is what I’d like to highlight here:

A frustrated Huang responded, “Because you’re out of imagination. For companies with imagination, you will do more with less. For companies that are, you know, when the leadership is just out of ideas, they have nothing else to do, they have no reason to imagine greater than they are. Then when they have more capability, you know, they don’t do more.”

As someone currently living through some of this and experiencing more and more consistent layoffs, along with “forced PIP’s” or even quotas for them, this really struck home.

He’s compared engineers’ usage of AI similar to how CAD is used by mechanical engineers - i.e., it’s a tool that you use to become more productive. So he’s definitely not an AI skeptic.

But I echo what he said here so much.

I’ve had so much pressure to simply “use AI” in my career lately that I genuinely feel like its required usage is missing the mark entirely.

What exactly are you doing with AI? Is it to simply use a tool or is it to use that tool to accomplish some task that furthers the goal of the company?

Is its usage making the customer’s experience better, however you define the customer to be?

Or is it just a way to pad your numbers and make it look like you’re a 10x engineer? And by extension, is this an attempt to make your company look like it’s doing more than it really is?

I think the simple, required usage of AI is the wrong metric.

It seems that a lot of companies these days, in their attempts to skirt from the repurcussions of the enshittification of their products, are using this as a scapegoat to throw their employees under the bus. And hopefully prop up their failing stock prices temporarily until their respective golden gooses are killed.

But that’s the next CEO’s problem, I suppose.

We’ve known for years that tracking developer’s commit and PR counts isn’t a good way to measure engineer productivity, why would this suddenly be different?

Today, though, besides the occasional one-liner, I think Perl is most known for its contributions to how we all (now) use regular expressions (this post isn’t going into what they are or how to use them).

However, there’s a bit more detail here that I’ve only recently filled in.

First, there was POSIX.

In the traditional Unix world, both “basic” (BRE) and “extended” (ERE) POSIX-based regular expressions were / are used.

There’s a bit more here than I’d like to go into, but in basic POSIX regular expressions, you have to escape parentheses with \( and use character classes like [[:space:]].

The most typical case for when you’d need to use these patterns are when using the unix shell command grep. You can also use the extended regular expression set by adding the -E flag or simply using the similarly-named egrep CLI.

Then, there was Perl.

Perl introduced another way to handle regular expressions, effectively adding some syntactic sugar that we’re all quite familiar with these days.

• \s for any whitespace character (or \S for a non-whitespace character)
• \d for a digit character (or a \D for a non-digit character)
• etc.

These are basically now the de facto “regular expressions” set that we use across all the major languages these days. Making this distinction I think is important, if only for historical purposes, especially since I always remember being confused why someone would use [[:space:]] instead of simply \s.

Enter the concept of “Backtracking”.

One of the most handy aspects to regular expressions are the usage of special references such as $1 or $2. These are called “backreferences” and refer to a previously matched set of characters within a defined set of parentheses. It can make things quite dynamic when looking through text.

I remember when first discovering them how much they felt like magic to me.

How we use them conveniently also helps us understand how the language’s runtime actually evaluates text against a given regular expression.

Backtracking is great. Until it’s not.

This is what I wanted to mention here - turns out there are two primary ways in which these regular expressions are actually evaluated.

• PCRE or Perl Compatible Regular Expressions
• RE2, originally a C++ evaluation algorithm created by Google

There is also a third option, the rust regex crate, which largely follows the same patterns and rules as RE2 does, as far as I understand it.

In PCRE, when the respective language runtime iterates over the bytes in a string to evaluate it against a supplied regular expression, it effectively runs through it as if it were a tree. This means traversing its branches, depth first, looking for what can be matched. When a “branch” doesn’t match the condition, the algorithm backtracks up a step and continues to a different branch.

However, in RE2, a state machine is created and each state can be evaluated, one at a time.

What does this mean?

Well in short, this means that RE2 does NOT support these backreferences and is slower than PCRE is. At least for smaller, simpler use cases.

This is where things get interesting.

If considering this as a “big oh” runtime algorithm, PCRE, while very flexible and awesome for most use cases, can suddenly be horrifically slow. It’s usually not, but it gets slow very fast as things get complex. And it can also exhaust memory and all of your cpu threads.

RE2, by comparison, is consistently linear in its big oh runtime. This means it’s “slower” for the simpler use cases by comparison, but is consistent and significantly faster and uses less memory.

RE2 is used in Golang, and it was a deliberate decision to do so.

I love to read the r/Golang subreddit and someone was asking why Golang’s regex pattern matching was “so slow” (they provided benchmarks). I was confused as to why my favorite language would be slandered in such a way.

But it got me deep diving into the topic and I learned some things!

What really helped me understand this better was a post-mortem write up.

I think the Appendix portion of Cloudflare’s write up / post-mortem from an incident they had in 2019 is the best way to understand more of all this. In short, they had a huge outage which was traced down ultimately to a complex regex pattern that, using PCRE for its evaluation, exhausted threads and memory.

I really liked what they wrote up on the topic, and they included some gifs showing just how many backtracking iterations can occur on even simplistic looking regular expressions.

The link to the appendix section: https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/#appendix-about-regular-expression-backtracking

It’s a bit dense, but it’s worth a read!

At least to some extent. I remember the content, but the context was a bit, dare I say unknown.

What we were discussing one day were HashMaps. We had previously discussed Linked Lists and a few other types, and I knew of the name “HashMap”, but what exactly were they?

In short, we were discussing:

• how to allocate an array of data somewhere in memory
• using a “hash function” to take an input parameter (a key) and “hash” it using a simplistic function (using the modulus function)
• locate that hashed location in our backing array
• either retrieving the value stored there or inserting one to it

The question inevitably came up - what if that location already has something in it?

Well, we’ll just move on to the next slot that’s free. Simple enough, right?

It all makes far more sense now, but I remember we finally came across the idea of using a Linked List to “chain” values in the slot instead of placing a singular value in the hashed location.

The idea was to help combat problems that arise when the hash table gets full enough. Suddenly looking for an empty spot is hardly any better than just starting at the first spot of the array and iterating over it one spot at a time.

So does this mean HashMaps aren’t O(1)?

I find that a lot of LeetCode problems generally just assume that all hashmaps are simply O(1) access / insert time.

How does that compare with any of the above, then? Shouldn’t it be O(n) big-Oh time for both reads and writes? Even reads have to search for the same hashed location, which may not immediately have the item you’re looking for.

Turns out this is a very well-known issue, and was actually a legitimate problem for languages like PHP and Java up until 2011 or so.

When hash maps get slow.

If an overly simplistic hashing function, like the one we did in in my Data Structures class, were used, we’d likely find ourselves colliding with other values rather quickly. And performing a secondary hash then can become a heavier operation on the CPU just to locate some data location.

What if you were unfortunate to the point where all items were hashed to the same spot in your hash map, requiring a linked list to store all of your values? Finding an item there would certainly be slow, as it would have to both hash the key, then go look linearly through the linked list.

Similarly, what if someone were actively trying to crash your system? Say they sent a bunch of similar requests to a POST endpoint, which caused a lot of collisions to happen, or possibly filled up your hash quickly, requiring the backing array to be resized over and over? This is a slow operation usually that requires all the data in the hashmap to be copied to the newly resized array.

Java, as it turns out, implemented a red-black tree data structure instead of using the linked list implementation to handle these cases. It also created a far more random hashing function than before, using SIPHash, implemented first in 2012. This, combined with the red-black tree instead of a linked list, made the max time when “chaining” occurs go from O(n) to O(log n).

This is significant for larger sized maps.

As seems to always be the case when it comes to computer science concepts, everything has trade-offs. A little bit of extra overhead will technically slow down small maps, but will significantly reduce the problems you can run into later. This is particularly the case when:

• the map fills beyond a 0.75 “occupied” ratio and data needs to be expanded
• keys get significantly larger
• the whole map gets large enough so it has to be allocated to the heap

Golang recently implemented Swiss Tables in v1.24+ (see the blog post) which is an additional extension to all of this, which is how I came across writing this post about it all. In it, they discuss breaking down a hash map into smaller tables, each of which contain bits that trade some bit space for information that avoids unnecessary chaining and similar look up operations.

It’s a fascinating read if you’re curious, especially since it has reduced map operation times for larger maps by as much as 60%!

There’s far more to consider there, but suffice it to say, a little bit of extra “record keeping” can significantly improve performance when things get large or get complicated.

Building a string seems trivial, and it generally is, but it’s a great way to dive into further to understand some of the basics around what makes Golang so interesting.

First of all, let’s note that a string in Golang is immutable. That means whenever you want to append to a string, the existing string plus what you’re appending needs to be copied to a new string in memory. This means that when optimizing code, we need to be mindful when we write code that performs these actions many times.

This is more of a common thing to think about in rust, but I digress.

In general, there are largely 4 primary ways to append to a string in Golang:

• Using string concatenation via the + or += function.
• Using the fmt module to use formatting directives.
• Using a bytes.Buffer to collect bytes that represent strings.
• Using the strings.Builder type from the strings library.

The last two are quite different from the first two. I find that they’re less often done because, well, people are either not as familiar with them or are more concerned about other, slower portions of their code.

Paying attention to why this is, though, is certainly helpful when writing other code that “smells” similar.

So by way of example, in the same order:

var str string

// string concatenation
str = "This " + "is " + "an " + "example"

// using formatting directives
str = fmt.Sprintf("I have %d examples", 4)

// using a byte buffer
var buf = new(bytes.Buffer)
buf.WriteString("This")
buf.WriteString("is")
buf.WriteString("an")
buf.WriteString("example")
str = buf.String()

// using strings.Builder
var bStr strings.Builder
bStr.Grow(4) // to pre-allocate memory
bStr.WriteString("This")
bStr.WriteString("is")
bStr.WriteString("an")
bStr.WriteString("example")
str = bStr.String()

In short, the fastest way of the above is the simple string concatenation use case. This is because during compilation, the Golang compiler actually performs a shortcut and makes this variable into a single constant, which is then referenced directly as a completed string from a special section of read-only memory.

There’s a caveat to this, however.

If you’re appending to a string within a for loop, the compiler often is unable to infer the combined constant size of the final string. This usually results in a bunch of extra copy actions and leftover memory that the garbage collector needs to come along later to clear out. Not great!

The slowest?

The slowest is the fmt.Sprintf() method, since it requires multiple calls to handle parsing 4 into a string type, then joins it together into a final string. It makes it incredibly easy for you as the developer, but it isn’t fast.

So how do each perform?

When running some benchmarks on all four options, the bytes.Buffer and strings.Builder methods are quite similar - still much faster than the fmt.Sprintf() method, but slower than the compiler-optimized concatenation.

Without including too much code here, using go test -bench=. on my local computer:

❯ go test -bench=.
goos: darwin
goarch: arm64
pkg: github.com/tmswfrk/stringbuilder
cpu: Apple M3
BenchmarkSB/WithPlus-8         	31813908	        37.52 ns/op
BenchmarkSB/WithSprintf-8      	 5284426	       228.4 ns/op
BenchmarkSB/WithBuffer-8       	15300552	        81.16 ns/op
BenchmarkSB/WithStringBuilder-8     16196134	        79.69 ns/op
PASS
ok  	github.com/tmswfrk/stringbuilder	6.369s

In the above, the columns are, from left to right:

• the name of the test
• the number of times that test was run
• how many nanoseconds occurred, on average, per allocation (to heap memory)

This means that the string concatentation method used the fewer number of allocations and ran the most times (due to its efficiency), while the formatting directive used the most allocations and ran the fewest number of times.

When should I use the strings.Builder or bytes.Buffer methods?

The strings.Builder and bytes.Buffer methods are really quite good when appending to a string and you don’t know many times you need to do so.

If we try to append using a for loop as mentioned earlier, you’re likely going to have a bad time. Let’s check out some example code:

func withPlus(x string) string {
	return x + x + x + x + x + x + x + x + x + x
}

func withPlusForLoop(x string) string {
	for i := 0; i < 10; i++ {
		x += x
	}
	return x
}

func withStringBuilder(x string) string {
	bb := &strings.Builder{}
	for i := 0; i < 10; i++ {
		bb.WriteString(x)
	}
	return bb.String()
}

When we test these using go test -bench=. you see some very interesting results:

❯ go test -bench=.
goos: darwin
goarch: arm64
pkg: github.com/tmswfrk/stringbuilder
cpu: Apple M3
BenchmarkSB/WithPlus-8         	31047177	        38.21 ns/op
BenchmarkSB/WithPlusForLoop-8  	  909802	         1234 ns/op
BenchmarkSB/WithStringBuilder-8     14542545	        74.36 ns/op
PASS
ok  	github.com/tmswfrk/stringbuilder	4.573s

Note just how much slower the string concatenation is with the for loop is!

True story, when attempting to run this from 0 < i < 1000, my computer basically froze. Crazy.

Where can I learn more?

I did most of this when attending GopherCon in 2025, but there’s a blog I’d suggest you read for more in-depth information: https://alexanderobregon.substack.com/p/go-string-concatenation-costs

When needing to present to an audience, instead of rambling on about what you did, lightning-talk fashion, it’s best to put some structure around it.

STAR is an acronym, pioneered (I think) mostly by Amazon for handling job interviews. But I feel like it’s helpful for any kind of technical presentation or interaction.

It helps you set the stage and build up details to a final sense of completion.

I really like MIT’s take on all of this, so give that a read for additional information than what I have here.

Situation

Set the stage. What key details should people know when considering the initial context around what was later done?

My company had X repositories being built by XX build tool, which had become increasingly difficult to reason about.

Depending on who you’re talking to, you may want to provide some measurable data behind a statement like this. Anything you provide here will really showcase your final result that you’re building to in this conversation.

Task

What was your task in this situation? What were you assigned to do, or what did you determine needed to be done?

My company saw this as an issue and I was tasked with creating / onboarding us to a new build tool.

I want to make an important distinction here - being assigned a task is very different than you being the one to set the task.

The latter showcases leadership. You saw an issue and actively chose to provide a solution to it, while also aligning key stakeholders and actions on the work. Perhaps another topic to discuss in more detail!

Action

How did you take action? What did you do?

This seems to be the most important part to the STAR method (some say 60% of the conversation), but highlighting it after the previous two helps resolve the “questions” you’ve set up previously.

I researched the market, got the budget approved, and allocated some local instances of XX build tool. Then I onboarded teams to using it by writing a distributed migration action…

This also needs to be catered to your audience, of course. If you’re talking to a peer, perhaps you need to dive more into technical detail. Maybe even get into some whiteboarding if that’s helpful.

But if you’re talking to a manager, focus more on the big picture of what was done. Let them determine where you go into more detail, and have those items ready in case they do. It helps a ton to know your audience with this one!

Result

So…what happened? Did you finish the action? Did it work?

We got everyone onboarded to the new tool and improved our build times by X% and reduced our costs by $XX.

It helps to have measureable numbers here, too, because it helps to normalize the actions to your audience. How much a dollar is worth is subjective person to person, for example, but we all know what a dollar is.

My take is this: STAR lets YOU set up your own questions and answers!

That’s how I see it, at least. And since you (hopefully) have a lot of situations that can be modeled after the STAR method, picking which one(s) to highlight is also an important skill in and of itself.

It’s a bit psychological, I suppose, but if you’re trying to highlight you as a candidate or your project as a potential for funding, controlling the narrative in this way is probably your best bet to getting the outcome you want.

How do you calculate the distances between two GPS coordinates?

Turns out it’s by using a function called the Haversine formula. And let me tell you, it looks kind of crazy at first.

I ride bikes a lot and use a GPS computer to track my rides - I’ve always wondered how a device, likely emitting data points every second or so - kept track of this information. And then additionally, how services like Strava or Garmin actually overlay that information on to a map to show you where you’ve gone, how fast, at what elevation, etc.

So I found some code references on how to use this formula, but it didn’t really make much sense to me. At least not until I found this video (which gets pretty heavy right in the last 30 seconds).

GPS Coordinates aren’t exactly like x, y coordinates.

You can do some basic calculations between two GPS coordinates when they share a common numerical value, but that’s not a very real-world example.

At first I thought, maybe I could treat this like a Pythagorean Theorem problem, until I realized that that assumption relied on things being in a flat plane.

And surely we have a better way to handle this than that. Earth isn’t flat!

The better way to think about this is calculating the “arc” that is created between two similar points. Then, you can see this as a trigonometric problem, using the Earth’s average radius as a data point.

With enough math, you can figure out this arc value “length”, aka distance. Basically you can use the trigonometric functions of a unit circle (with Earth’s radius as its “unit”) if you imagine taking a slice of the Earth at the longitude you’re comparing.

Remember radians? I had nearly forgotten!

Where things get more interesting though is when you want to compare two GPS points from anywhere.

The Haversine formula takes this into account. And it does so by imagining a similar “unit circle slice” but oriented differently, i.e., NOT along latitudinal or longitudial lines.

Really cool! The video above explains it a bit better and with visuals.

So I decided to write up a small Golang function to calculate this value between two data points.

Using a type that includes both a Lat and Lng value to combine values into a singular “point”, it looks a bit like this.

type TrackPoint struct {
  Lat float64
  Lng float64
}

func haversine(t1, t2 TrackPoint) float64 {
  // Earth's approx radius in meters
  const earthRadius = 6371000
	
  var (
    // radian calculations
    rLat1 = t1.Lat * math.Pi / 180.0
    rLat2 = t2.Lat * math.Pi / 180.0
    
    // delta values, converted to radians
    dLat  = (t2.Lat - t1.Lat) * math.Pi / 180.0
    dLng  = (t2.Lng - t1.Lng) * math.Pi / 180.0
  )
	
  // account for going over from -179 to +179 degrees
  // a simple subtraction would make this look like -358 degrees
  if dLng > math.Pi {
    dLng -= 2 * math.Pi
  } else if dLng < -math.Pi {
    dLng += 2 * math.Pi
  }
  // this is kind of like a mod function that helps 
  // when numbers wrap around a boundary, except it
  // +/- two radians / 360 degrees.

  // the math
  a := math.Pow(math.Sin(dLat/2), 2) + 
    math.Pow(math.Sin(dLng/2), 2) *
    math.Cos(rLat1) *
    math.Cos(rLat2)
	
  c := 2 * math.Atan2(math.Sqrt(a), math.Sqrt(1-a))
	
  // turns out this is equivalent to above, and its been
  // way too long since Calculus 2 for me to know why
  //c := 2 * math.Asin(math.Sqrt(a))

  return earthRadius * c
}

Which can then be called via a main function:

func main() {
  // Big Ben in London (51.5007° N, 0.1246° W)
  // New York (40.6892° N, 74.0445° W)
  // Expected distance: 5574.840457 km
  var (
    london = TrackPoint{Lat: 51.5007, Lng: 0.1246}
    nyc    = TrackPoint{Lat: 40.6892, Lng: 74.0445}
  )

  fmt.Printf("Haversine distance between NYC and London: %f\n", 
    haversine(nyc, london))
}

Which then shows us:

Haversine distance between NYC and London: 5574840.456849

I find things like this particularly fascinating because, despite its complexity, it’s something that’s used in every day functions without anyone really realizing it.

“Enshittification” is the gradual degradation of online platforms, where services initially provide high-quality, free, or cheap value to attract users, then pivot to prioritize advertisers, and finally extract all value for themselves, ultimately harming both users and business customers.

I love this term.

It describes so much of what I’ve seen in tech over my lifetime. Seriously great products and processes enabled by the “world wide web” of the mid to late 90’s, eventually to succumb to ads, paywalls, popups, and “nickel and dime” schemes in both physical and online products.

If you’ve ever wondered by Outlook is now a subscription service instead of a standalone product that you’d buy in CompUSA, I mean, Best Buy? This is why.

I don’t run ads here, partly in protest to this trend.

My websites that I run are served very efficiently from blog storage and I refuse to run ads on them that hinder your experience reading them. It’s my response to this general market trend in hopes that I can align myself with old men yelling at clouds, smelling new books in a local book store, and oogling over some LP’s.

There are stages to “enshittification”.

Cory describes “enshittification” in 3 stages.

Stage 1: Good to Users. Platforms use subsidies or excellent service to attract a large user base, creating a “lock-in” effect.

Think about how Facebook gradually took over from MySpace, for example.

The requirement to have a college email address made it feel more exclusive compared to MySpace, which additionally drew people in.

I was in college right as it was coming online, and everyone else had a Facebook account, why wouldn’t I, a freshman in college, use it? Besides, MySpace felt so high school.

And with all of your friends and family on Facebook, why would you leave?

Stage 2: Abuse Users for Business Customers. Once users are locked in, the platform degrades the user experience to favor business customers (advertisers, sellers) to generate revenue.

At some point in 2009, I got addicted to a dumb game on Facebook called “Mafia Wars”. I even created a second Facebook account to try to build up my “mafia” more quickly.

It “gamified” my time, which meant that I had to log in every few hours to perform some actions, then wait for reset timers to cool down.

Later, I got blasted by Farmville requests from other friends (and family). I even enjoyed the Simpsons: Tapped Out for a while, too (which was admittedly hilarious in some ways).

Little did we realize that ads were creeping in, and microtransactions were happening so that we wouldn’t have to wait as long for those timers to reset.

Besides, what’s a few dollars for a game you spend a lot of time on?

It’s just so satisfying to have that one special edition character roaming around your virtual Springfield. At least that’s what the dopamine told me.

We as human beings started to become the product for our technical shareholders.

Stage 3: Abuse Business Customers for Themselves. Finally, the platform squeezes the business customers, taking nearly all the value for shareholders and executives, leaving a broken, less usable platform.

Enter today. What do we see most at Facebook these days? I’ll share what I see:

• Posts shown to me that encourage me to get angry, because I’m more likely to interact with them.
• Misorganized posts and threads that make context absolutely devoid of meaning, creating content that’s shouted into the void.
• Ads that are disguised like posts and show up when scrolling anywhere.
• Feed algorithms that sometimes deprioritize the things I’d like to see.
• Occasionally something good, usually over in the Marketplace side of things.
• etc.

There was even legal action in this space, specifically the Cambridge-Analytica data scandal, where political campaigns were altered due to the way these algorithms worked, often in nefarious and orchestrated ways.

Having BMW charge you per-month to use your own heated seats was a big step into the world of enshittification, too.

But since law takes longer to litigate than “innovation” iterates, we are regularly left behind by the drastic and ill-conceived actions of a few tech companies. They pocket the money and run, then broker a reduced set of fines that usually don’t reflect the real impact they’ve left on us.

So anyway…

Probably the best way to read / learn about “enshittification” is directly from the source on YouTube, at CloudFest 2025: https://www.youtube.com/watch?v=_Ai-fC-2Bpo

He even went on the Daily Show, which was hilarious and amazing to watch.

Up next, Norway?

I wanted to mention it in today’s blog post because even Norway has gotten in on it: https://www.youtube.com/watch?v=T4Upf_B9RLQ

It’s a wonderful, tongue-in-cheek take on that whole situation we face ourselves in today. Give it a watch!

Perhaps with enough time, we all can refocus our attention away from the tech billionaries running the show now and back to ourselves, our friends, our family, and our own lives.

One can dream!

Mainly because I’ve personally found it to be more confusing than it should be.

It usually looks a bit like this, when following a worker pattern:

func main() {
  var doneCh = make(chan struct{})

  // Perform some concurrent work
  // ...

  // Wait on that work
  for i := 0; i < 10; i++ {
    <-doneCh
  }
  
  // complete
}

I intentially left out a bit here to highlight what a “done” channel actually looks like.

Following up on the worker pattern post yesterday, I also connected a few dots that previously I had never connected.

A done channel does effectively the same thing as a WaitGroup.

In the example above, the <-doneCh command is a blocking operation. The command waits for something to be sent to the channel before it can read an empty struct from it. In this case, a send operation is performed at the end of a function being called from within a go routine.

Note that the content of the channel doesn’t matter - only that something is sent to it. It’s only there to block further operations while longer tasks finish their work.

The struct{} type is a little bit special in Golang.

Bare struct{} types in Golang allocate no memory. This is super helpful if you want to build a Set or something where you only care about the presence of a key, or in this case, the presence of anything in the channel to gate behavior.

From Dave Cheney’s blog and using unsafe.Sizeof() to show memory usage of items, we can see that a struct{} takes no bytes.

var s struct{}
fmt.Println(unsafe.Sizeof(s))
// prints 0

So rather than using up more memory than you need to storing throwaway data, it’s a great idea to use chan struct{} for a done channel.

Going back to the example above…

Using the previously discussed worker pattern:

func main() {
  var (
    workCh = make(chan int)
    doneCh = make(chan struct{})
    numWorkers = 10
  )

  // set up a worker pool
  for i := 0; i < numWorkers; i++ {
    go doWork(i, workCh, doneCh)
  }

  // add task items to our work channel
  for i := 0; i < 100; i++ {
    workCh <- i
  }

  // close after adding all tasks
  close(workCh)

  // block using the doneCh
  for i := 0; i < numWorkers; i++ {
    <-doneCh
  }
  // complete!
}

One more important thing to add here is how the doWork() function operates!

Similar to how, prior to Golang v1.25.0, we had to pass in the WaitGroup to the go routine’s function to decrement it, we have to do the same here with the doneCh.

func doWork(wid int, workCh chan int, doneCh chan struct{}) {
  for work := range workCh {
    // ... do something that takes a while
  }
  
  // send an empty struct to the channel
  doneCh <- struct{}{}
}

(See the Go Playground link for the complete example)

Note the similarities to a WorkGroup!

Previously, we would:

• wg.Add(1) prior to calling the go doWork() operation.
• Pass in the wg to the function.
• defer wg.Done() to signal “done” when doWork() was completed.

Whereas in this pattern, the exact same process occurs:

• Set up a blocking call on the done channel with <-doneCh.
• Pass in the doneCh to the function.
• Send an empty struct to the channel when doWork() was completed.

Just like you have to wg.Add(1) for each worker’s invocation of its go routine, you have to set up a blocking “wait” on the doneCh.

Using a done channel without the worker pattern

Without using the worker pattern, you need to model this function using a single worker in-line function:

func main() {
	var (
		workCh = make(chan int)
		doneCh = make(chan struct{})
	)

  // set up a single worker
  go func() {
    for work := range workCh {
      // do some work until channel is closed
      // ...
    }
    // signal the end of the work tasks
    doneCh <- struct{}{}
  }()

  // send all tasks to work channel
  for i := 0; i < 100; i++ {
    workCh <- i
  }

  // close when all tasks are sent
  close(workCh)

  // wait for the signal
  <-doneCh

  // complete!
}

(See the Go Playground link)

This is a good pattern to use if each task is simple and fast to perform, but adding each task is expensive.

All this being said, I’d still recommend using the sync.WaitGroup method if possible, while also utilizing the newer constructs available in Golang >= v1.25.0, since it’s just far easier to reason about!

But this is still great to know, and I know I’m glad I went back to review this one again.

API calls, especially when each can be handled separately and independently, are easily implemented with this pattern, and I’ve seen it help tremendously when downloading artifacts for building in CI, for example.

Instead of setting up a buffered channel with a capacity of 10 in an attempt to concurrently handle 10 requests, you can create an unbuffered channel and read from it using 10 workers.

To do this in a coordinated fashion, however, it does require the usage of a WaitGroup, which is part of the sync package of the standard library.

Let’s take a look:

func main() {
	var (
		wg         = new(sync.WaitGroup)
		ch         = make(chan int)
		numWorkers = 10
	)

	// set up a worker pool
	for w := 0; w < numWorkers; w++ {
		wg.Go(func() {
			doSomething(ch)
		})
	}

	// add work items to our channel
	for i := 0; i < 100; i++ {
		ch <- i
	}

	// close the channel after adding everything
	close(ch)

	// wait for our workers to finish
	wg.Wait()
}

func doSomething(ch chan int) {
	// read from the channel
	for item := range ch {
		fmt.Printf("Doing some work for item %d\n", item)
		time.Sleep(time.Millisecond * 500)
	}
}

(Go Playground link)

Pretty sweet, isn’t it?

WorkGroup usage prior to Golang v1.25.0 is different.

Note that in versions of Golang prior to v1.25.0, the usage of a WaitGroup was slightly different. Instead of using:

for w := 0; w < numWorkers; w++ {
	wg.Go(func() {
		doSomething()
	})
}

It would look more like this:

for w := 0; w < numWorkers; w++ {
	// add an item of work
	wg.Add(1)
	
	// also pass in the wg to the func
	// and call as a go routine
	go doSomething(wg, ch)
}

Which also required you to pass in the WaitGroup to the function like so:

func doSomething(wg *sync.WaitGroup, ch chan in) {
	// signal done to the wg
	defer wg.Done()

	for item := range ch {
		//...
	}
}

I really like the new wg.Go() function, since it abstracts away the slightly awkward wg.Add(1) and wg.Done() function calls.

Worker IDs in output can help with debugging.

You can also pass in the worker id to the function being called for additional traceability. Then you can show which worker is doing which item of work.

func doSomething(wid int, ch chan int) {
	// read from the channel
	for item := range ch {
		fmt.Printf("Worker %d did work on item %d\n", 
			wid, item)
		time.Sleep(time.Millisecond * 500)
	}
}

Check out the Go Playground link if you want to see it in action.

You can actually see concurrency in action by noticing the numbers scolling by “out of order”. Each request is going to take a different amount of time, and you can rely on the Golang runtime to handle all the context switching for you.

...
Worker 1 did work on item 67
Worker 9 did work on item 65
Worker 5 did work on item 68
Worker 4 did work on item 66
Worker 0 did work on item 69
Worker 2 did work on item 70
...

It helps to have a “unit of work” defined as its own function.

For readability and maintainability, I recommend having the actual “work” relegated to its own function.

And importantly, have that function be simply defined as a single unit of work, rather than comingling it with concurrent parameters (unless you need mutexes, for example, to gate certain non-thread-safe actions).

This makes the calling function essentially a wrapper that handles the concurrency. It’s kind of a “separation of concerns”.

It also makes for easier unit testing - another topic for another day!

Do not communicate by sharing memory; instead, share memory by communicating.

They’re awesome because they enable concurrency support “out of the box”.

They’re basically their own type that implements concurrent safe logic of sharing memory locations of data without the explicit usage of mutexes or locks. The example in the blog link above has some code blocks that show this quite well, actually.

They can be buffered or unbuffered.

This is implied in the way you define each type:

var unbuffered = make(chan int)
var buffered   = make(chan int, 3)

A great way to describe the difference is that an unbuffered channel will block the go routine it’s being called from until it’s “filled” with something.

You can also specify their “direction”.

This means you can make them “bidirectional”, “send”, or “receive”. Note the locations of the <- identifier.

var bidirectional = make(chan int)
var sendOnly      = make(chan<- int)
var receiveOnly   = make(<-chan int)

You can combine these attributes together.

So you can have any combination of buffered, unbuffered, send, receive, or bidirectional.

var sendOnlyBuffered  = make(chan<- int, 10)
var receiveUnbuffered = make(<-chan int)

You can change this in function headers.

So this means you could have the following:

func main() {
  // create a bidirectional channel
  someChannel := make(chan int)

  myFunc(someChannel)
}

// specify channel in header as receive-only
func myFunc(ch <-chan) {
  // ...
}

Not only does this convey meaning, but it can make sure that your accidentally try to write to a channel that you’re only expected to read from. Especially important if you have a bunch of different go routines doing different things!

Reading from channels is done via the range function.

A few additional points about this before an example:

• to read from a channel, you can use the same range function similar to how you read from a slice.
• due to this blocking behavior, reading from an unbuffered channel is often done within a separate goroutine.
• it’s always a good idea to close the channel when you’re done using it, usually via a defer.

func main() {
  // unbuffered
  var ch = make(chan int)
  defer close(ch)

  // see below
  go doSomething(ch)

  // populate the channel with items
  for i := 0; i < 10; i++ {
    ch <- i // "send to" the channel
  }
}

func doSomething(ch chan int) {
  for item := range ch {
    fmt.Printf("Item was read: %d\n", item)
  }
}

(see Go Playground link)

This just iterates over a set of numbers as if it were a slice, but it showcases that channels can work similarly. While also allowing for concurrency and without needing to create and manage your own mutexes. Neat!

How how would an unbuffered channel handle a longer task?

Where things get interesting is when you simulate longer running operations in the above code.

Say you modify it a bit to include a time.Sleep statement to simulate a worker action (an api call, a db transaction, or something similar):

func doSomething(ch chan int) {
	for item := range ch {
		fmt.Printf("Item was read: %d\n", item)
		time.Sleep(time.Second) // simulate long process
	}
}

(another, separate Go Playground link)

When running this, you’ll notice that the “send to” action isn’t gated by anything in particular.

Yet we only read one item from the channel per second!

This means that the first for loop that populates the channel is being blocked because there’s no “buffer” in the channel to queue later iterations up for later.

Enter buffered channels, which have a capacity.

This means that if you create one with a capacity of 2, but then try to send it 5 things, 3 of them will be blocked until the first two are read.

func main() {
  // make buffered channel with capacity 2
  var ch = make(chan int, 2)

  // same as before
  go writer(ch)

  // simulate slow process before reading
  time.Sleep(time.Second * 2)

  for item := range ch {
    fmt.Printf("Read %d from channel\n", item)
  }
}

func writer(ch chan int) {
  for i := 0; i < 5; i++ {
      ch <- i // this will block on 3, 4, 5
      fmt.Printf("wrote on iteration %d\n", i)

      // show delay between reads
      time.Sleep(time.Second)
  }
  close(ch)
}

(see Go Playground link)

So yeah, channels are pretty great. But they can be a bit tricky to grok at first, given that they look and act a bit differently than other parts of the Golang ecosystem.

Either way, expect to hear more from me about channels soon!

Everyone knows that debugging is twice as hard as writing a program in the first place. So if you’re as clever as you can be when you write it, how will you ever debug it?

Brian W. Kernighan is attributed to that quote, and it (regularly) reasonates a lot with me.

I think sometimes as engineers, we actually like complexity. It feels fun to build something really interesting and technical in scope, and we often confuse simple topics like “simple”, “technical”, or even “beautiful”, “clean”, “elegant”…the list goes on.

I actually also like this variation on the same quote:

“Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.”

Supposedly by Martin Goldin (although I can’t find a definitive account of which Martin Goldin this is).

It’s only gonna get worse with AI.

I don’t know if you’ve heard, but there’s this thing called “AI” and it’s supposed to make life better for all of us and automate us all out of existence.

I’ve seen it being used a lot lately, with engineers priding themselves on how they don’t actually need to write code anymore, relying on AI agents like Claude, ChatGPT, Gemini, etc to do it for them.

Nevermind the actual “engineering” part of all this.

Regardless, the more coding you offload to an agent to do on your behalf, the less you know about how it actually works.

And the more complicated it’s going to be to figure things out when they (inevitably) go wrong.

Unless you very proactively review it and fix these PRs as they come, of course. But given how often we all see "LGTM" on our Github PR’s, I have a feeling that this may very quickly spiral out of control if we don’t manage it.

So let’s take a step back.

If AI works for you and it simplifies your life, great. But be aware of what it’s doing.

Otherwise, that same agent you used to create that cool feature is going to become your crutch to debug and fix things when it goes wrong.

So when you’re paged at 3am to fix a site outage, you’re going to be leaning pretty heavily on your prompt engineering skills, along with maybe a few thoughts and prayers from your asleep coworkers, to fix the issue.

Is that better or worse than it is today? Time will tell, I’m sure.

I’m referring to azcopy sync.

I sometimes use this to copy over files from the generated public/ folder that Hugo creates for me to my Azure blob store. That’s where this site is stored.

In another post, I wrote about a handy VSCode plugin that I use when uploading changes to this site. Unfortunately, this one deletes everything and re-uploads everything.

A bit much, right?

Well, the alternative here is to use azcopy sync. It involves a few things…

Download the azcopy binary

The best place to do this is to locate the right binary for your OS + Architecture from the main Azure page: https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10.

Generate a SAS key for a certain amount of time

Do this within the Azure portal. In the $web container within your storage account, you can generate a SAS token using the ... button to its right.

From there, define whatever time interval you want for this token (shorter is better, as is the case with cryptography) and optionally set it against your IP address that you want run the sync operation from.

Make sure to give it the right permissions, usually read, write, delete, and list, especially if you plan to use the --delete-destination flag.

It’s important to keep track of this key!

Once you navigate away from this page, you won’t be able to see the key again. You’ll instead have to create another one - luckily, there’s no limit to these apparently!

Run the right command

Make sure you follow this pattern:

azcopy sync "<local-directory-path>" "https://<storage-account-name>.blob.core.windows.net/<container-name>"

If you’re using this at all like I am, you’ll need to escape the $ character from the $web container that prepends all of this. Or just use single quotes as opposed to double ones.

And you’ll also need to make sure the full SAS key is added to the URL, too.

So in total, something like this.

azcopy sync '/Users/<$USER>/code/some-hugo-project/public/' 'https://mystorageacct.blob.core.windows.net/$web\?sp=racwdli&st=2024-11-21T06:24:19Z&se=2025-11-21T14:24:19Z&sip=123.456.789.987&spr=https&sv=2022-11-02&sr=c&sig=<someOtherGobbletyGook>'

Fun!

The irony in this is that I’m 100% going to just use the VSCode plugin for pushing this change. But once the site gets bigger, I’ll definitely need to set up a new SAS key and run the sync command instead.

You don’t technically need a SAS key to do this, btw.

I just find it easier. Using azcopy login will create an “Azure AD” or “Entra ID” and that can be used more directly rather than creating a whole separate SAS token. With the way my subscription is configured in Azure, I’ve found logging in like this to be a bit cumbersome, so I just use the SAS token instead.

If you’re unfamiliar, this was the shuttle that blew up on its way up out of our atmosphere. I remember less of the actual event myself, but was well aware of it because my elementary school was named after Christa McAuliffe. She was to be the first teacher in space.

What does this have to do with management, you say? Take a read from NPR:

[Boisjoly] found disturbing the data he reviewed about the booster rockets that would lift Challenger into space. Six months before the Challenger explosion, he predicted “a catastrophe of the highest order” involving “loss of human life” in a memo to managers at Thiokol.

The problem, Boisjoly wrote, was the elastic seals at the joints of the multi-stage booster rockets. They tended to stiffen and unseal in cold weather and NASA’s ambitious shuttle launch schedule included winter lift-offs with risky temperatures, even in Florida.

It continues:

On January 27, 1986, the forecast for the next morning at the Kennedy Space Center included a launch-time temperature as low as 30 degrees Fahrenheit. NASA had never launched in temperatures that cold and Boisjoly and his four colleagues at Thiokol headquarters in Utah concluded it would be too dangerous too launch.

Three weeks later, he told NPR’s Daniel Zwerdling in an unrecorded and confidential interview, “I fought like Hell to stop that launch. I’m so torn up inside I can hardly talk about it, even now.”

I live my life in software, and it literally never has any real impact to human life, so I cannot even fathom how this man must have felt. That quote is just…difficult to read, let alone comprehend fully.

In a well-running organization, it’s our duty to communicate risks appropriately.

I’ve recently been through a lot professionally. As a “developer’s developer” (my self-inflated title), I’ve found myself working tirelessly in the realm of DevX to provide a better set of tooling for my (mostly Golang) developers.

Unfortunately, I’ve often struggled to communicate and align with management over various priorities.

In a well functioning organization, I firmly believe that management listens to its engineering staff.

In DevX and internal tooling, engineers are effectively “customers” in the retail sense.

Management should be asking them:

• What do they struggle with day to day?
• What processes would you like to see simplified or added?
• How well do they understand the existing build and configuration behavior across their supported environments?

It’s pretty rare to see a management chain that is as familiar with the day to day technical efforts of the engineers they oversee. In my experience, this is often driven from a “top down” or “bottom up” approach. Both require good communication, which I think is the point I’m getting at here.

Having a top-down approach can often be dicey, and not having good communication makes it a recipe for disaster.

Having said this, however, engineers aren’t always right, either. “Bottom up” approaches without the right workforce can also lead to a lot of technical challenges, mostly discovered later.

But if you:

• Do the right hiring
• Listen to those employees on the ground doing the work
• Dogfood the code that you’re supplying to your engineering workforce

…you’ll likely find yourself working well within an organization that values its technical workforce.

Managing people is hard enough - why make it more difficult by refusing to communicate?

Getting back to Roger Boisjoly…

If you don’t speak up, management won’t know, and the organization will suffer.

If you do speak up and management doesn’t listen, the organization will suffer.

Strangely enough, I’ve found that a lot of engineers these days don’t feel empowered to speak up. Tech has become cutthroat and with nearly constant layoffs, people generally just want to collect a paycheck and do other, hopefully more interesting things with their lives.

But sometimes, just sometimes, people speak up and others listen.

And things get better.

One can only imagine if management had listened to Roger on January 27th, 1986 and had chosen to postpone that launch.

func (t *Trie) collect(n *TrieNode, pfx string, results *[]string) {
  // ...
}

Where I started to spiral was the *[]string type in the final input parameter.

That’s weird looking, isn’t it?

In Golang, slices and arrays are often confused. You may say: “But aren’t slices copied by reference?”

And you would be wrong. Like I was. Everything is copied by value in Golang! Even when it appears like they aren’t.

Lets look at some source code…

A quick look at the runtime/slice.go file shows this as the slice data structure:

type slice struct {
	array unsafe.Pointer
	len   int
	cap   int
}

What does this mean, exactly?

• An “array” is the underlying type / struct that stores your data.
• A “slice” is a wrapper struct that includes both the data you’re storing and some metadata about that data.
• A “slice” is technically a “slice header”.

This is why when you instantiate an “array”, you need to provide it an explicit size, which basically is a type on its own:

var arr1 [4]string // creates 4 empty strings
var arr2 [3]string // different type!

Compare this to the far more familiar form(s):

var s1 []string
var s2 = make([]string, 3) // requires an initial size
var s3 = []string{"something", "goes", "here"}

Getting back to my “gotcha”…

When you pass a slice to another function, you most typically pass it as a []string. It’s easy and it looks nice if I don’t say so myself.

And since most functions are written to return another []string or possibly something else entirely, it’s pretty rare that you have to think about what’s really going on here.

Let’s look at a contrived example:

var s = []string{"something", "goes", "here"}
fmt.Printf("original: %s\n", s)
modSlice(s) // function call, passes s by value
fmt.Printf("after return: %s\n", s)

Where modSlice() is defined as:

func modSlice(s []string) {
	fmt.Printf("inside func orig: %s\n", s)
	s = append(s, "another")
	fmt.Printf("inside func modified: %s\n", s)
}

Should be fine, right? You expect the new entry, “another”, to be included in the final “after return” print statement.

Perhaps not:

original: [something goes here]
inside func orig: [something goes here]
inside func modified: [something goes here another]
after return: [something goes here]

(see Go Playground link)

It looks like the new entry isn’t actually writing to the backing array.

So why did this happen / not happen?

If you list out the s variable from both the caller’s perspective and from the function’s perspective, you should effectively see the same item:

Caller: mySlice{array: 0x123, len: 3, cap: 10}
Function: mySlice{array: 0x123, len: 3, cap: 10}

After all, the function just received a copy of the slice (header). The array attribute includes the same array reference.

When we add a new entry to the slice via the append function, mySlice becomes larger in len while modifying the same memory behind the array parameter:

mySlice{array: 0x123, len: 4, cap: 10}

However, when we return back to the caller’s stack and context, our mySlice variable still has:

mySlice{array: 0x123, len: 3, cap: 10}

Note the 3 in the len parameter!

When fmt.Printf receives the s slice in our original example, it only knows to print up to 3 items, effectively ignoring what does indeed exist in memory at the additional position we just added!

Gemini’s response

Earlier I noted that Gemini told me to pass the slice by ref as *[]string.

This was to guarantee that when the append function was called inside the function, that it would modify the same slice. It would do the same memory manipulation as before, but would also update the len and cap struct attributes of the same slice, too.

More can be said about why this isn’t the best idea (more allocations to the heap, confusing function signatures), so I rewrote my code to avoid this whole situation to begin with. But for completeness, if you were to make this change to the above contrived code:

func main() {
	var s = []string{"something", "goes", "here"}
	fmt.Printf("original: %s\n", s)
	modSlice(&s) // note the reference
	fmt.Printf("after return: %s\n", s)
}

func modSlice(s *[]string) {
	fmt.Printf("inside func orig: %s\n", s)
	*s = append(*s, "another") // both refs
	fmt.Printf("inside func modified: %s\n", s)
}

And then run it, you would see:

original: [something goes here]
inside func orig: &[something goes here]
inside func modified: &[something goes here another]
after return: [something goes here another]

(see Go Playground link)

Each have enabled all kinds of development from small to medium companies so that they don’t have to set up a bunch of servers, letting them sit idle most of the time.

Much better than having Bob accidentally knock the power button on your production box under his desk, then lock the door and head home for the weekend.

That being said…

This site is hosted on Azure.

There has been a long standing program through LinkedIn where employees get a credit each month if they set up and use Azure. So that’s what I’ve been using to host this site and my other one, bicyclewatercooler.com.

The short of it is this:

• I write blog content in the form of markdown (.md) files.
• I compile them locally with Hugo.
  • I do this via hugo --baseURL https://www.nilpointer.blog/ as it’s easier to keep localhost in my hugo.toml during development.
• I upload the generated public/ directory to my configured Azure blob store.
• Sometimes I need to purge the CDN / FrontDoor configuration so that the new assets I upload are reflected to the public.

VSCode + Extensions works well for this.

With a few plugins, I can easily upload the generated content directly to my static website storage. There are two extensions that are needed for this (at least I think both are needed):

• Azure Resources
• Azure Storage

The only “gotcha” here is that the plugin operation that uploads your files to your storage has to delete everything and reupload everything. Not a big deal for a site that can manage plenty of downtime (“it’s just a website!”), but it gets annoying after a while. There’s a way to do this via an rsync like operation, but I’ll save that for another post.

Overall, this all works quite well.

Mainly because you don’t have to set up any kind of web server to serve your content. You can just serve static contents via Microsoft’s extensive CDN.

With small, non-dynamic content, it’s quite efficient and basically costs pennies to serve each month.

If I ever have to move to a new provider, I’m sure it won’t be too difficult to start up something similar in one of the other providers!

Hot take: the code used in infrastructure engineering shouldn’t be complicated, nor should it be clever. It actually should be rather dumb.

If something goes wrong, you want it to be with the code you’re trying to build or ship, not an unexpected effect of an overly complex CI system.

Delegate to native tooling everywhere you can. Don’t be clever. Be clever elsewhere and save yourself the maintenance effort.

I got a bunch of comments on the topic, so I guess labeling it a “hot take” was apropos.

Why is this Controversial?

In my experience, CI is quite an abstract topic, even for software engineers. But without proper guardrails, good intentions can go awry and get out of control quickly. Especially when you have distributed actions across multiple Github Actions configuration files and/or CLIs.

Some good takeaways that I liked:

“Most things can be solved in a simple way.”

Occam’s Razor approves this message, as do I.

Kernighan’s Law “Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.”

Absolutely love Kernighan’s Law. Debugging in inherently hard. Doing it in a highly customized and “environment-assuming” environment that intentionally isn’t set up like your dev box is even harder.

What I keep seeing with teams is that every time infra code gets “smart”, it’s usually compensating for missing decisions upstream. Then six months later no one knows why the pipeline behaves the way it does.

Couple this with “the road to hell is paved with good intentions” and you have a recipe for all the custom infrastructure that keeps DevX engineers in business.

Doing the Digital Janitor Work

I recently had to sort out some GHA actions (is that redundant?) to fix some compatibility tests across some Go modules. It was initially configured in one way that never quite worked - until another engineer implemented a command-line find | xargs command to locate N other go.mod files.

Super abstract. But it DID work. One-liners are fun!

Diving into the issue, I found that in a previous setup step, we were zipping up some source files in a way that included their full, absolute paths. This meant that when extracting them in the other job, we suddenly had a bunch of duplicative paths that looked an awful lot like the previous path it was all starting from. Gross.

No one had bad intentions here, mind you, and I could have just continued dealing with it on my end. But I decided to fix the issue upstream, sorting out the zip action to better handle a shared Github runner.

I didn’t need to, but leaving it in that state and then continuing to do my job would have made it far more difficult to reason about later.

These kinds of things happen in software engineering.

They just do. And I imagine that AI may make this worse.

But can we all agree that simpler, “dumb” CI is better when it comes to building and deploying code?