TTT 21: The Gray Zone

I'll be sharing a thing I did, a tweet I loved, a thought I had 💭

Summi, Vigi, and I in front of Vigi’s restaurant, New Asha Sri Lankan Restaurant, in Staten Island. More about it lower, keep scrolling!

Hi friends,

இரவு வணக்கம் (iravu vaNakkam), Buenas Noches, and Good Evening from New York City!

I was inspired by Louie Bacaj and Christine Trac to make the move from Ghost to Substack for the TTT Newsletter. The network effects of Substack’s recommendation engine pushed us over 100!

I’ll be sharing a technical “How-To” in next week’s newsletter for any Ghost bloggers here :)

Tweet I Loved

This tweet lives in my head rent-free

Imagine getting woken up at 4am on a Sunday morning by a page from your app.

The app is getting a persistent error message from the platform. The platform team says everything looks good on their side. The platform team asks for app team for metrics.

The app is still down.

The app team has 15 minutes to get the app back up, otherwise they violate the Service Level Agreement (SLA) to the Product Team. 

A violation could mean a health professional makes a clinical decision based on stale data.

For me and the teams I work with in pharmacy, like an update to a patient’s allergies.

At most companies I’ve worked at, the platform teams would assume the issue is on their side and work together with us (the app team) to mitigate the issue. Sometimes the issue even ended up being on the app side.

Most of these companies were small.

Recently I’ve moved to a much larger company. There are more teams managing more fractured platforms. Each team has a narrower scope.

Between each of these scopes are the interaction points between the app and the various platform teams.

This is called “The Gray Zone”.

Where “The Gray Zone” is for my visually inclined friends.

And unless the app team can ensure that next time the error shows up these metrics are available in 1 click, the SLA will get violated again.

Persistent violations of the SLA will erode trust between the product and engineering teams.

Without trust, the company cannot build the future. Building the future is incremental, and relies on tinier extensions of “the benefit of the doubt” along the way.

At smaller companies, the platform teams were so engaged keeping the SLA felt achievable.

A platform team would keep a log of times they helped different app teams. They’d compile database connection pool settings for all incidents apps had. They’d suggest optimal configs given the app team’s network topology.

They had a memory.

This same platform team would have admin privileges to run packet captures to debug connectivity between IPs.

Sometimes the platform team was proficient in promQL and could write queries to get metrics from your app.

This mostly lets app developers stay app developers.

With a smaller number of teams on the production bridge, the overhead of team coordination alone gave the app team a chance to mitigate the outage within the SLA.

But, all of this “going above and beyond” is not within the traditional definition of running a platform.

Acknowledging that “The Gray Zone” exists is the first step to improving operations in distributed systems.

The next step is defining what each team must bring to the “The Gray Zone”. 

It’s the only chance the app team has to meet tight SLAs.

Tweet I Loved

Cam Houser is in Quito, Ecuador September 21-25 @cahouser

Twitter bios are limiting. Here's a 60 second video that explains: -who follows me (entrepreneurs and knowledge workers) -what I teach (video) -why they follow me (to grow their companies and their careers)

I always overthink my video content.

It can be so easy to forget to introduce yourself to your followers. This is a dope template of one way to do it. I’m gonna imitate this.

If you’ve also forgotten to introduce yourself to your followers, give it a try and let me know how it goes :)

Thing I did

Last week, my friend Summi and I captured some footage for a Youtube video at a Tamil restaurant, called New Asha Sri Lankan Restaurant, in Staten Island!

When we walked in, the owner, Vigi, started speaking in English.

I took a big gamble by replying in Tamil. I was nervous because I don’t usually speak with strangers in Tamil. Summi jumped in and his Tamil was impressive!

We spent the next 3-4 hours speaking in Tamil and sharing tasty bites.

Fried Chicken Biriyani (weekend special). Egg Roti, Goat Curry, Passionfruit Juice. This is what you should order if you go :)

At first the words were all in my mouth like marbles.

But eventually I calmed down and my Tamil started flowing. Another stranger in the store even said my Tamil sounded smooth and sweet.

Summi said he’s going to post the video on Youtube October 1. I’ll be sure to share it here whenever he posts it :)

I recently realized that what I do on a daily basis to improve my Tamil is what needs to be shared. It’s a multi-pronged strategy. On an average day, I spend about an hour across various habits.

Whether that’s 

1. Vocab review on Anki (every day)

2. Listening to Hello Kekutho (3x/week)

3. Watching Jaffna Suthan (3x/week)

4. Calling Amma (Mother) or Aru Periamma (Aunt) or Giri Mami (Aunt) (1x/week for 45 minutes)

5. Doing Mimic Method exercises with Hello Kekutho or Jaffna Suthan (1x/week for 20 minutes)

6. Speaking with strangers (3 hours last week in New Asha restaurant)

7. Studying grammar and verb conjugations to add tenses and texture to my speech (haven’t done much lately - want to get back into this)

I learned a hyper-specific regional dialect of Tamil from the Northern Province of Sri Lanka while never having left New York City. 

That blows my mind.

And if I can do it, you can do it too.

---

I'd love to hear any feelings you felt while reading this and until next time - be easy. 

Love,

Janahan