The Digital Domino Disaster

Jenny: Hi friends! Welcome to today’s episode of We Are, Marketing Happy–A Healthcare Marketing Podcast. My name is Jenny Bristow and I am your host. And I’m the founder at Hedy & Hopp, a full-service, fully healthcare marketing agency. I’m here today joined by a Senior Digital Producer at Hedy & Hopp, our very own Suzie Schmitt. Suzie, welcome. 

Suzie: Thanks, Jenny. Glad to be here. 

Jenny: We today are going to be talking about something that most marketers have been frustrated with over the last year, but most of them don’t have a solid tech understanding of the whys, either through the lens of, why is this happening? Or, what can I do to help us, our organization, moving forward? And that is: tech dependencies. And the Jenga tower, that is the internet that holds up all of our, ecosystems in not only health care, but really across the web. I mean, all of us have been burned by the recent Cloudflare or AWS outages over the last year. It took down the entire internet, it felt like, for quite a long period of time.

So, Suzie, give us an overview, kind of as marketers, you know, see something like AWS go down or Cloudflare go down and everything stops working. Why is that happening? 

Suzie: That’s a great question. And it’s one that I’ve been asked increasingly from our clients, given those outages you’re talking about. And really what’s happening is that when we are clicking around in the internet and we’re browsing what we see as the web, we are abstracted so many layers away from the underlying infrastructure, and we’re really just only seeing that tiny little tip of the iceberg of everything else that’s behind that.

So we’ve got a whole bunch of things going on, which is one, we have enormous monopolies that own giant chunks of the internet. So that’s why when AWS goes down, it feels like the entire internet has gone down. Because even if a website doesn’t use AWS as its hosting provider, it probably has something on the site that runs on AWS.

And generally, people don’t even know all of the dependencies they have. You know, a website can have hundreds or even thousands of dependencies, and you might have a plug in and that plug in itself, you might think, well, that’s one dependency. That plug in itself probably has 50 dependencies all by itself. So if one of one tiny thing breaks across the internet, it can have enormous butterfly effects and like a ripple consequence throughout the entire internet, with Cloudflare being out, we noticed that not only was everything unreachable, but also so much of the security wasn’t able to be authenticated, so people weren’t being able to browse to web properties even if they could connect to them, which most people couldn’t. 

So the reason this impacts us more in healthcare is that unlike other industries, we’re held to uptime and other SLAs by CMS. So we have to be really diligent about planning for not if, but when these things go down and how to mitigate that risk. 

Jenny: Yeah, really great point. Let’s make this real with a real world example. It’s been nearly a decade since the left pad incident. Suzie, what is the left pad incident? 

Suzie: Jenny, I’m so glad you asked. I could talk about the left pad incident. All day. The left pad incident was in 2016, there was a copyright dispute between an independent developer out of Turkey and the messaging platform Kik. This developer had released a software package on NPM, which is the node package manager, which essentially, if you’re building any software, you’re going to be using that to say, well, I don’t want to build a whole framework to said, put a table in my website, I’m going to use this prebuilt module of this table and then be able to fill it out. It’s basically a library of basic functionalities. Anyway, he had made left pad, which all it does is add padding to the left side of text. It’s 11 lines, but theoretically like very minor, but it was built into so much of the internet as a dependency or a dependency of the dependency that when he got mad at the copyright situation and simply unpublished his package, it took down thousands of websites, including enterprise websites like Netflix and Spotify. They went down, all because of one open source, volunteer, unpaid developer in Turkey. So in that instance, NPM actually did something completely unprecedented and they republished his package. 

But ever since then, it’s been a little bit more top of mind to if you’re going to use something as a dependency, like maybe make your own fork of it, or make sure that you know how it works. If it’s something really simple, have your own copy. But it really does highlight how one tiny little spoke holding up the rest of the system can just be taken away, and it can really go a lot of problems. 

Jenny: So I want to talk about this in two lens. First, I want to talk about cloud monopoly risks. Then I want to talk about open source devs.

So let’s start talking about clouds. There’s really just three companies that host most of the web. Right? Let’s talk about that. What does that mean?

Suzie: Absolutely. So AWS which is Amazon Web Services, they host over a third of the internet. I believe last time I checked it was about 33%. Azure’s lower than that. And then GCP is a little bit lower than that. But together those companies host the majority of websites. And also the majority of enterprise websites. And what’s really scary there is generally when you choose a hosting environment, that’s also the hosting environment that your backups are created in. So you know, and I know here at Hedy & Hopp that we have a third party disaster recovery backup system that is hosted on a completely different infrastructure than our day-to-day production environment. But if you are hosting on AWS, for example, chances are that if you make a dev site or a backup site or any of those images of your website, they’re probably also hosted on AWS. So therefore, when AWS goes down, everything goes down. You don’t have a backup because your backup is also down. So the smartest thing to do is to keep a backup somewhere that’s completely isolated from wherever your production environment is, just to not keep all your eggs in one basket. And to make sure that those backups stay up to date. 

Jenny: Yep, that’s wonderful. What about Suzie—explain, in non-technical terms, what Cloudflare is and how that impacts accessibility of sites. 

Suzie: Definitely. So Cloudflare is something in between the website and the user. It’s like a little layer. If you think of a website as the moon, this is a satellite somewhere in between Earth and the moon, and you’re not actually connecting to that website in the moon, you’re talking to that satellite because the satellite says, hey, we’ve already got a copy of this website that’s recent, it’s closer to your location, and we can authenticate that you’re a secure user. So we’re going to go ahead and feed you this cached version. And generally what that results in is a more secure and faster internet. And when it works as designed, it works fantastically.

But when it doesn’t work well, what that means is not only can nobody reach the moon, but also that copy on the satellite is gone. And also whenever the moon is back up and running, the satellite has to rebuild all of its copies. So if you noticed that the internet was running slowly, not just during, but after the AWS outages, it wasn’t just you. That’s because all of those caches had to be rebuilt. All of these systems depend on each other, and when one breaks, it has a domino effect. 

Jenny: Yeah. So there’s really multiple potential single point of failures. Whenever you think about the marketing tech stack in the digital ecosystem that, you know, we’re creating here in the healthcare space. Let’s talk about open source devs and how they are essentially holding up the internet for free.

Suzie: That is correct. And it’s an unsustainable position we found ourselves in. So as we all know, the internet kind of started and then, like most great sandboxes for innovation, it just exploded with no real rules or guardrails. And what that is resulted in is a foundation that’s made of, out-of-date, fundamentally insecure programing languages like things built on C at the infrastructure level, that then that means that everything built on top of that is just a piece of duct tape to try to patch the infrastructure.

It’s like if you had a foundation from the 1800s and you tried to build a modern house on top of it. It’s—the plumbing doesn’t match, the electric doesn’t match, everything shorts out, things aren’t compatible. So we’re dealing with that every day. And we’re also relying on people who are working on it as a hobby in their spare time, unpaid, with no real motivation other than a self-felt sense of satisfaction to maintain really fundamental things that then billion dollar enterprises depend on every day. 

So it’s a really interesting dynamic where nobody’s really responsible to fix it. And it’s an enormous issue that would take an incredible amount of resources to truly resolve. So everybody is just kind of building their own duct tape towers, essentially. And we are just going to find out what happens. 

Jenny: So, really great analogies, by the way, making it very easy to understand for our non-technical listeners. So thank you, Suzie. Let’s now make it actionable, right? I hate ending podcasts or sharing a bunch of scary information and then saying, have a good day. So we’re in healthcare. We have very specific rules around compliance and security. So really what folks need to do is make sure that their organization has a disaster plan. So, Suzie, what advice would you give to internal teams to create and then be prepared to execute a disaster plan?

Suzie: Definitely. I mean, with all of these systems out of your control, it’s irresponsible not to have a disaster plan. You have to assume that one of these vendors is going to fail you eventually. And when that happens, it’s better if you just can follow your guidelines and react calmly instead of trying to build the plane while you’re flying it through a fire.

So the first thing I’d say is make sure that you’ve set all these internal rules and that everybody’s aware of their role and feels comfortable supporting that. You don’t want to say, oh, well, we’re going to have Jenny handle that. And then when it comes to pass, Jenny’s like, oh, I don’t know what that means or I don’t—I’m not comfortable speaking to that. So you want to make sure that you have all those roles and your SMEs organized and a chain of command and a chain of communication. It’s always important to make sure that you keep those notifications going. So if something is wrong and you’re a covered entity, it’s important to say that something is wrong, even if it’s not your fault, even if it’s a global outage that you had nothing to do with. And it’s important just to communicate something, even if you don’t have all the details like, “We are down. This portal is inaccessible. To our knowledge, your data is safe. We will keep you updated. Our next update will be at this time.” People are generally pretty understanding as long as you give them something to work with and provide a reasonable amount of transparency.

So I’d say make your plan and, stick to it. Maybe even do a test run, you know, see how it lands, how the language lands with your account managers. I’ve done that, to say, do you think this would be acceptable or should we maybe explain this a little bit more? So it really just depends on the services you offer. But gotta have that disaster plan. 

Jenny: Yeah, disaster plan is great. I want to reiterate your key point about making sure your backup is on a different, server or ecosystem cloud provider than production. Super important, right? One thing that we need to remember for folks that are covered entities is with HIPAA. Breach notification timeline starts ticking when the outage happens, even if it’s not your fault.

So you need to make sure that you have a really clear documentation of timeline of events, and then work with your security or privacy officers to make sure that you really understand the next steps. But you have all the documentation covered. And then finally, I love your, understanding and like the explanation of roles, but also making sure that the messaging is super clear, right? Have drafts ready to go, you know, for some of these scenarios that, you know, are likely to happen at some point. That way you can execute upon it really quickly. 

And then finally, if somebody wanted to and they should, right? How would they go about documenting their dependencies? A lot of organizations don’t even know, you know, what all their dependencies are. How would they go about trying to create, you know, that documentation to layer into a disaster plan? 

Suzie: That’s a great question, where you’re really going to want to start is the services you support in-house versus the ones that you outsource to a vendor. And anything you support in-house, you should be able to list all the dependencies, or if you don’t, the only acceptable reason is that you have a third-party vendor that supports something that has a proprietary system, but you have a clear understanding of who is your point-of-contact on their end in case of a disaster. 

But beyond that, you should be looking at let’s just start with maybe your website that you manage in-house. Take a look at every plugin on that site. If it’s WordPress (It probably is. Most of them are.) And then you can go ahead and you can look at that plugin. Most of them have really robust documentation. Most softwares have the requirements and dependencies file. Read the read me file. I’m one of the very few people that does that, but it’s almost always very helpful and you can get a pretty good list. I would say it’s tedious, but not technically difficult. And it’s something that you should have because if you see one of these common vulnerability exploits go out, you need to be able to answer the question very definitively of whether or not it affects you. Because if it does, to your point, that breach notification timeline, the clock is ticking. 

Jenny: Yeah. And I just want to remind marketers that, you know, many of you may be thinking in your head, oh, this is something IT will handle. Well, should they handle it? Maybe. But just like all of the things around compliance and security, it’s now another hat that marketers are having to wear. So definitely, if you think your team is handling this, make sure that you meet with them, confirm, get copies of all the documentation that they’ve created to make sure that you are aligned with the plan and you understand what will happen next. Because honestly, the communication should fall to the marketing communications team, not IT. So make sure that you’re working in lockstep with them.

Suzie: And I will say that when we made our disaster recovery plan, we learned right there a lot of the roles. So a lot of the questions that you might say, I don’t even really know where to start. Go find a good example of one and start following it. And before you know it, you’ll be adapting it to your unique products and systems. It just, it happens. So and I will say also, that made us realize how many roles we actually needed. So it’s important to know the, the lift. You know something should it happen. It’s better to know ahead of time. So.

Jenny: Absolutely. So awesome. Well, Suzie, thank you so much. This hopefully will answer a lot of questions and help all of our listeners be prepared for the inevitable next outage that will happen in the internet.

And thank you, as always, for tuning in. Share this episode with a team member that you think may find value and like and subscribe. We drop episodes almost every Friday. That is our goal. And if you have a topic you’d like for us to cover, shoot us a note: hello@hedyandhopp.com. We’d love to hear from you.

Until next time, thanks for joining us on this week’s episode of We Are, Marketing Happy. Stay safe, friends.