683

October 23rd, 2023 × #Database Failures#Site Outages#Data Exposure

Spooky Coding Horror Stories 2023 - Part 1

Developers share horror stories of catastrophic mistakes like deleting databases, breaking site functionality, exposing data, and more.

or
Topic 0 00:00

Transcript

Announcer

Monday. Monday. Monday. Open wide dev fans. Get ready to stuff your face with JavaScript, CSS, node modules, barbecue tips, get workflows, breakdancing, soft skill, web development, the hastiest, the craziest, the tastiest web development treats coming in hot. Here is Wes, Barracuda, Boss, and Scott, El Toro Loco, Oh, syntax.

Announcer

On this Monday history treat, we'll be sending you to the depths of

Scott Tolinski

Am I talking about spooky stories? We're talking about Spooky dev stories. This is one of our annual episodes where we do where we talk about spooky stories. I'm not gonna do the spooky voice the whole time, but This is part 1. This is the hasty version of spooky stories that we're gonna be telling on the podcast where basically people write in. They talk about Spooky dev stories.

Scott Tolinski

Dev stories that are very spooky, spooky, spooky, spooky stuff like dropping the whole database or maybe charging 10,000 users at, You know, who knows? My name is Scott Tolinski. I'm a developer from Denver, and with me as always is Wes.

Wes Bos

Wes, oh, I like that. Yes. Thank you. Freestyle. Alright. So we have 2 shows for you. This is gonna be the first one, a little hazy. Just a couple. We're going to get you wet your whistle.

Wes Bos

And then on Wednesday coming out, we have the big one, which has some some doozies in it. You certainly don't want to miss that one either. These are going to be fantastic stories of Wanting to stick your head in the sand, ways people have goofed up in tech, deploying, deleting databases, all that

Scott Tolinski

awful, awful stuff. We should talk about what you do if you're scared, because maybe you're frightened. You it's Spooky Saturday, and you pushed a code on Friday, and, you don't have error in exception handling tracking. You gotta have that. So check it out at century.i0.

Wes Bos

And, yeah. Just just let them know that you you heard about it from Syntax. That'd be that'd be pretty rad of you. So let's get into it here. Alright. First1 we have here, my dev horror story was I was only 1 month into my 1st job as a junior web developer. These are always the most awful ones. There's so many that are, like, on my 1st day on the job, my 1st week on the job, I was working on-site updates for one of the largest furniture retailers in the country.

Topic 1 02:34

Junior dev deployed JS without null check, breaking site

Wes Bos

I was still very inexperienced and wrote some JavaScript. We worked on the page I was making updates to. However, I had no null checks in the JavaScript bundle to run on every other page. So if the element didn't exist on the pages, JavaScript would break.

Wes Bos

Very common thing. You got to check if the element exists there before you go ahead and use it. And of course, it runs if the element's there. But if that JavaScript runs on another page, thing will break. Somehow this code passed PR review in internal testing as they were only testing the page updates had been made on.

Wes Bos

Interesting.

Wes Bos

So the code was deployed to production completely breaking JavaScript across the site right before the retailer's biggest Online sale of the year.

Wes Bos

What's worse is the deployment to production happened on a Thursday, and this issue was not Notice until late Friday afternoon.

Wes Bos

Uh-huh. I realized what had happened and told my manager, but I was a junior developer, and my manager said I must be wrong. There's no way something small as a missing null check could have caused a What issue? I was told to go home so I can't stop laughing and that others would have to stay late on Friday. That's not funny at all to attempt to resolve or roll back a deployment.

Wes Bos

I never was blamed or called out for causing the issues, but the guilt of causing the team members to work late on a Friday night for my mistake still haunts me to this day. Oh, brutal, brutal, brutal. Hopefully, it was just a, like, quick little if statement in there, but Who knows? Yeah. Woof.

Scott Tolinski

Alright. Next one. The embarrassing test page incident. Many years ago in my 1st web developer job, published a test page with a silly message to a WordPress site. Folks, keep your silly messages for for Slack. Never. Never. I've I've done it myself many times. They will get out to to real people. We we've all done it. Yeah. Little did I know, the website Site had a live Twitter feed that displayed the latest tweets from the company's account. To my horror, the test message appeared on the feed Visible to 12,000 followers.

Topic 2 04:20

Accidental embarrassing test page published to live site

Scott Tolinski

Although I promptly deleted the page, the tweet remained.

Scott Tolinski

We had We had to seek help from another SEO firm to remove the embarrassing tweet. Oh, dang, folks. You gotta get control of your own Twitter. I think it's a little bit of an own goal here that you can't delete tweets from your own Twitter account.

Wes Bos

Yeah. No kidding. That's wild. Yeah. Wow. That is Crazy. Yeah. You think those things are kind of funny where they set off a chain chain reaction. You know, you think, oh, I'll just delete This thing and everything will be fixed. But those things unravel and they go like even sometimes we Publish a podcast before it should be out.

Wes Bos

Do we hear about it, man? Like, even if you turn it off, There's it's still in caches around the world, and they still ripple through. And it's one of those things where it sets off a chain reaction. It can be impossible to put the genie back back into the bottle. Yeah. I was still getting messages about the episode that didn't work, like yesterday even after you had fixed it. Yeah. The Quest oversight. As a lead developer for a web established SaaS product, my role included reviewing and deploying pull requests to production. During a busy period, I Received numerous Pro requests with substantial code changes.

Topic 3 06:09

Code review oversight led to login vulnerabilities

Wes Bos

One particular request had extensive occasions, spending multiple files and lines of code.

Wes Bos

Feeling overwhelmed, I briefly skimmed the change and tested them on my end. Satisfied with the results, I merged and deployed the request. However, that's much better than the last one we read, which was LGTM.

Wes Bos

Looks good to me. Well, a few days later money.

Wes Bos

We received a support ticket reporting a concerning issue.

Wes Bos

Users were able to log in with incorrect passwords and even access other users' accounts.

Wes Bos

Tips. Upon investigation, I discovered that the authentication code had been mistakenly altered. The developer responsible confessed that the disabled passwords Do we do to forgetting their own? Oh, man. Yeah. Uh-huh. Sometimes Sometimes you think, you know what? I I'm working on this feature. Let's just disable this other thing that gets in the way so I can focus on the feature.

Scott Tolinski

Don't forget. I fixed it. Pushing it high.

Wes Bos

Oh, from that point on, I implemented stricter review protocols and sought assistance from the team to prevent such oversights in the future. Honestly, that is such a human mistake that the only thing that can save you from that is automated test of testing every single make sure that it logs in. Make sure that it incorrect password doesn't work. Make sure that a blank password doesn't work.

Wes Bos

Oh, woof.

Scott Tolinski

That's a bad one too because that take a straight up security incident. The email is gonna be going out.

Scott Tolinski

If this is, like, a a public company that's, you know, money off the stock price or whatever, could you imagine the the value That would actually have an a real, like, big scenario. Yeah. Luckily, hopefully, this was a a small scenario a small time company.

Scott Tolinski

Alright. Next 1 is this is a developer horror story from my 1st professional role. I was tasked with making changes to a JSON file, essentially a lookup table We're integrating a learning management system with a student management system that would later pass data to our national education registry.

Scott Tolinski

LMS to c SMS to NER.

Topic 4 08:23

Cleaning duplicate JSON data caused education issues

Scott Tolinski

I I intentionally didn't read this one because I could Tell that it was very well put together,

Wes Bos

so I didn't read this one. And I'm cringing at the very first thing, which is 2 systems put together Pushing data to a national registry.

Scott Tolinski

Yes. Right. Yeah. You see the word national in any of these, and it, definitely gets your ears perking a little bit.

Scott Tolinski

We had a script that converted a CSV into a structured JSON tree that our integration used to determine what national Competencies was to be awarded to learners as they progress through their degrees.

Scott Tolinski

My responsibility was to add new competencies to this table for learners.

Scott Tolinski

Without a complete understanding of how it worked, I noticed duplicate entries in the lookup table. I discussed this with a colleague, and together, we decided to remove them entirely to clean it up in quotes due to the substantial Increase in lines of both the CSV and JSON files. The differences between the 2 became significant.

Scott Tolinski

When it came time for my review, my manager Glance it over without much concern. Alright, folks.

Scott Tolinski

Fatal flaw number 1 here.

Scott Tolinski

I decided to clean this up Without fully understanding what it does. That's like looking at, like, a box full of wires and then, like, I'm just gonna pull all these Yeah. In your does it. Yeah. Probably nothing.

Wes Bos

Yeah. Hey. Probably nothing.

Scott Tolinski

Double data? I I don't think it's running anywhere. I'll just unplug it. Alright. I mean, I don't need it. That's like when I build something, I have all these extra bolts Probably for nothing. Just yeah. Who needs them? Right? Okay. We gotta clean this up. So we pushed it to production. After another review, I Celebrated my 1st professional poll request had been deployed.

Scott Tolinski

8 months later, I received an email from our client stating that over 3,000 students had incorrect competencies awarded to them, and some even had their degrees aspire to missing information.

Scott Tolinski

Oh, gosh.

Scott Tolinski

As my colleagues and I tried to unravel the mystery, we discovered that the system actually required those duplicates to award historical competencies to learners.

Scott Tolinski

Okay. So the duplications had a meaning. Right? Believe it or not.

Scott Tolinski

Reverting the changes And reintroducing the duplicates to the lookup table was straightforward, but now we needed to write a script to review approximately 60 50,000 individuals in the system ensuring they had received the correct national competencies for their degrees.

Scott Tolinski

As it turns out, about 7,500 learners nationally were affected.

Scott Tolinski

We created this script to simulate these updates, Had our client review and approve the differences and then nervously initiated the integration to correct the students.

Scott Tolinski

After processing about 500 students, we encountered a rate limit issue.

Scott Tolinski

After negotiation with the systems provider, they temporarily lifted the limits allowing us to proceed.

Scott Tolinski

After in the end, we resolved the issue. I didn't get fired, though my managers were understandably frustrated.

Scott Tolinski

They admitted they should have provided more oversight.

Scott Tolinski

Regardless, It was an exhausting few weeks working around the clock to rectify this monumental screw up. I have to say discovering that your Pull request was the root cause of the problem after opening a ticket searching through the logs and finding the issue Yeah. Was the worst feeling I've had in my professional career.

Scott Tolinski

On the surface, it might not seem like a big deal because we fixed it, but the worst part about incidents like this is knowing that your code caused significant disruptions to other people's work weeks.

Wes Bos

Yeah. Yeah. I'd say that's And, like Like, possibly people's lives. You know? Like, if someone's degree Expired? You know, like, you you plan or or if you're like, oh, do I have all the credits? You log in. Oh, I got I got them all, and then they're clawed back.

Wes Bos

You're like, oh, well, I just moved to Denver. I can't go back. I can't go back. I had a a nontechnical

Scott Tolinski

scrub. My My, adviser in college once told me that I had enough credits to graduate, and then, like, 2 months before graduation, he was like, oh, oopsies. You were you're missing this one thing.

Scott Tolinski

You're missing. So I did a a special, like, a a project where I I created a bunch of music for him, and he just gave me the credits I needed. So it worked out.

Wes Bos

But some of these stories are like, I broke something, and we immediately realized how awful it was, and some of these stories are like, we realized on a slow burn, you know, over 8 months. This next one I'm all done with it. Slow burn.

Wes Bos

I deployed a disallow everything robots dot txt. So robots. Txt is the file that search engines use to say what pages am I allowed to crawl.

Topic 5 13:14

Disallowed search engine crawling tanked rankings

Wes Bos

I didn't manage analytics, so it went unnoticed for 3 months. This is for a huge ecommerce platform, and it tanked Their search rankings for a while after. Oh, you cannot quick fix that. That takes forever To come back from.

Scott Tolinski

Oh, that it and, yeah, messing with search search rank can be devastating for a number of reasons. And like you said, it's not a quick fix. You can't just call up Google and say, oopsies. I made a whoopsies, and, can you fix this for me? Yeah. You're you're Definitely a big trouble at that point. You know what?

Wes Bos

I'll I'll tell you right now for everybody looking for lessons learned on this one is sign up for I think it's like Google search indexing or something like that. Yeah. And they will email you anytime web messages, anything anytime anything weird comes up. Like, I got an email this morning of a website that I have. I have, like, a little demo video on one of my websites, and it's, like, Like, 500 pixels by 300 pixels. This is tiny. Right? And I got an email from Google saying we can't index this video because it's too small, And I went, oh, like, that's helpful to know. Like, I don't want it indexed. It's not like a video that should go in as a video thing. It's just A video for the website to show a demo, but,

Scott Tolinski

yeah, that's that that's good to know. Yeah. Holy cow. Alright. Next one. I had worked On implementing GDPR deletion of customers for a few months and was ready to launch.

Topic 6 14:55

Wrong AWS deletion command wiped customer files

Scott Tolinski

One of the things to delete was Customer uploaded files in AWS s three.

Scott Tolinski

I meant to delete files starting with, for example, customer forward slash 123, Where 123 was the customer ID, but I accidentally forgot the trailing slash.

Scott Tolinski

People who are experienced can see where this is going. Right? This means that not only files starting with customer 123 were removed, but files Starting with customer 1234 and 12345 or anything after, etcetera. In other words, deleting 1 customer could remove files from other customers.

Scott Tolinski

Be because, yes, they weren't checking the end.

Scott Tolinski

Luckily, we had another problem with GDPR Deletion, so we had to roll that back quickly.

Scott Tolinski

A few days later, we found out about the wrongly deleted files. Thanks to that early rollback, Only about 400 files were deleted wrongly instead of tens of thousands, a blessing in disguise. Okay.

Scott Tolinski

You know that you messed up When you can say that accidentally deleting 400 files was a blessing in disguise because the alternative is is is much greater than that. It's 10,000.

Wes Bos

Yeah. This next one comes from, again, somebody who I we can't say. We keep them all anonymous, But this person was worked for some very large fang type companies.

Topic 7 16:22

Deleted Postgres database, team lost access

Wes Bos

I dropped the backing disk for the production Postgres, Then fell horribly sick. So basically, you deleted entire Postgres deleted a hard drive, which had a Postgres database on it.

Wes Bos

While I was unable to respond on Slack that whole weekend, the rest of the team realized their credentials were on my desktop, And they couldn't deploy without contacting Amazon support.

Wes Bos

Mistake number 2 is Only having 1 person with the magic keys, I came in still a little sick on Monday to a bunch of tired faces and dozens of DMs on Slack and no clue what had happened.

Scott Tolinski

Yeah.

Scott Tolinski

Being blissfully unaware.

Scott Tolinski

Next one. I accidentally pushed staging code as an update through code push to production Android app of my company.

Scott Tolinski

Gave me a production Android app of my company.

Topic 8 17:17

Pushed dev code to production Android app

Scott Tolinski

The rollout was also to 100% of users. Hey. Nice. Because you can do a, stage rollout on Android like that. I'm sure you can on Ios. Luckily, we realized this soon as in the production testing, users were automatically to be logged out, And the user does not exist in the staging database. Token return error. I checked the branch name and it was developed.

Scott Tolinski

I still remember this moment.

Scott Tolinski

We immediately halted the rollout and pushed a new update with the production code to all users. It was the worst day of my life as developer.

Scott Tolinski

Only me and the QA leader are aware of this, and thanks to her, she had not told this to the morning stand up the next day. The next day, I wrote a Bash script that double checks the git branch before pushing an update to production environment. And since then, nothing like this has happened as the script Access and intermediate. Yes, folks. This is why people often have, continuous deployment, continuous integration setups because They do this stuff for you. It's like, yeah. If you push to main, it will run production. Now I don't understand exactly how that would work in an Android world. I don't do that kind of stuff.

Scott Tolinski

Yeah. Pushing production pushing dev to production is definitely something I think a fair number of us have done at some point in our our careers by accident.

Wes Bos

Next one here. Biggest tech fail. I forgot to take out a test condition and sent around 10,000 emails to 200 people each. Oh.

Wes Bos

Wait. That's 10,000 times 200 people is I don't know how much, but a ton Wait. Ton of emails. 10,000 times 200? Oh, I know. I know what that is.

Topic 9 18:44

Forgot test condition, sent mass emails

Scott Tolinski

What is it? It is 2,000,000. No. It's no. It's not. 10,000 times 200.

Scott Tolinski

Is it 2,000,000? 10,000.

Scott Tolinski

Oh, yeah. I was thinking Oh, no. Try to call me out for math, mister. 1, 2, 3 times 200. It is 2,000,000. Now I am mister Magoo. I got the big old mister Magoo award.

Wes Bos

Never try to do math in front of people. You always make a

Scott Tolinski

Make a pool of yourself. Make a mister Magoo out of your

Wes Bos

Next one. A previous employer, we made white label mobile app. So White label means where, like, you make, like, an app that's, like, kind of generic, and then different companies will use that app and just brand it themselves as whatever they want. One of the features we had a calling a configured phone number. Well, 1 client was a religious group with a toll three number for a prayer line.

Wes Bos

One day, they emailed support to change it. I guess someone misread 8884 866 for the area code. And instead of a prayer line, it went to an adult hotline.

Topic 10 19:53

Misconfigured phone number sent to adult line

Scott Tolinski

Oh, wow.

Scott Tolinski

That's a great that's a the exact wrong place you'd want that call to go.

Wes Bos

I didn't know. Maybe they had some support there. You know? Yeah. Right. Yeah. No one admitted the mistake, and we didn't try, Hard enough to find anyone who had done it. Another more technically consequential one was accidentally clearing out a varnish cache Across every single instance, which had thousands of large RSS feeds cached, this served millions of requests daily.

Wes Bos

Back end wasn't happy. Yeah, because a Varnish cache will RSS feed might be a massive database query, So you cache the results of that. So maybe that query runs once every 10 minutes, or maybe it runs once Hey. Who knows? Right? But if you if you clear them all and they all need to rerun at the exactly same time, you're probably gonna run out of CPU or some resources, or you're gonna get a bill that is massive from your cloud service. Anyways, he has he goes on to say related to RSS feeds, you may or may not know, but if you change the GUID, GUID, guaranteed unique identifier for an item, it will cause clients to automatically redownload the associated file.

Wes Bos

This is kind of we want to move our podcast RSS feed, but I need to look into that because That scares me of waking up one day. Somebody says, hey. I got 600 episodes of New episodes. Yeah. Yeah. Well, aren't we just we're just reserving

Scott Tolinski

Our hosts.

Scott Tolinski

Oh, you're saying if we move our host, if we move our podcast. If we move our host. Yeah. Yeah. Totally. Something I'm sure guarantee we're not the 1st people in the world. Luckily, we own the RSS feed. We just have to look into it. Yeah. And and what I I know is that at least the some of those services that we're looking into have specific you you tell them what service you're coming from, and we will Hopefully, white glove migrate you. So Yeah. If you accidentally change every

Wes Bos

guaranteed unique identifier on Very popular feed that lists a ton of large audio files. It turns out that your entire back end dies a horrible death, and Amazon sends you a huge

Scott Tolinski

Bill. It was just like you said, both of the, both of the instances there. Oh. That's hilarious.

Scott Tolinski

Oh, man.

Scott Tolinski

One time, I ran an update statement on a production ACH database making all of the routing and account numbers bogus.

Scott Tolinski

We had to restore the backup from a previous night, and the data entry team had to key in all of the HCH, ACH info again.

Scott Tolinski

Needless to say, they didn't like me anymore.

Scott Tolinski

Yeah. Your solution to the problem is having to Key stuff in by hand, and if you have, you know, 1,000 users if you have more than a 1000 users I mean, even if you have a 100 users, that's That's like a awful time. Then Yeah. What? You can break that into task for a certain amount of people to do, but, gosh, anything that ends up in manual data entry, Brutal. Not my favorite type of thing. We should say for non Americans,

Wes Bos

ACH is automatic clearinghouse.

Topic 11 23:20

Garbage account data required manual reentry

Wes Bos

That is how people in the US send Money between accounts and banks, right? So not only was this a major screw up, But, like, it's it's people's money at and, like, at stake here. You know, you don't wanna accidentally send that to the wrong person.

Scott Tolinski

Yeah. Right? Can you imagine manual data entry of these bank account numbers, and then you accidentally you know, you you just goof up looking at the wrong?

Wes Bos

Yeah. Frog wine. I once made a, the first time I ever did stickers, I had to wire The printer $10,000 to print them all.

Wes Bos

And don't at the time, my bank didn't have the ability to do a wire online, So I had to take the invoice and the payment details from the printer.

Topic 12 24:13

Incorrect wire transfer account nearly lost $10k

Wes Bos

And I walked to the bank and I sat down and it was just like, I had never seen this teller very young, and I sat down and I said, hey. Like, I need to wire $10.

Wes Bos

And I was wiring it to China. So, like, I wasn't wiring this to another Canadian bank or maybe in the States. I was wiring it to a bank that you could not read the name of.

Wes Bos

And so she I give her the piece of paper. She sits there and punches all the numbers in and hits the submit button.

Wes Bos

And then all of a sudden, I don't like, a couple days later, I asked the printer, hey. Did you get the money? And they said, no.

Wes Bos

We didn't get anything. Like, we contacted the bank. Mhmm.

Wes Bos

And so so I took the The little piece of paper that they give you when you make a wire transfer at the bank. Yeah. I I scanned it in. I sent it on. Like, look. This is proof of payment. He said that's the wrong account number. So I looked at it, and I go, oh, no.

Wes Bos

I just sent $10 Oh. To a random and a Chinese bank account that I do not know.

Wes Bos

So I went into the bank. I was like, look.

Wes Bos

You typed this number wrong, and this Poor girl was shaking.

Wes Bos

You know? Luckily, the bank sent it back.

Wes Bos

Like, it took, like, a week and a half. So there was, like, a week and a half of just, like, did I just send $10 into the Yeah. Into the ether? It's gone forever. It's gone forever, But luckily the bank returned it and everything was fine. But the next time I had to go in, I got the same teller and she was you could tell she was just like, Can you triple check this? You know? And we sat there, and we're like, 7, 4, 3. They're making sure every and, like, it's so funny because a copy paste solution would have fixed that. Although, in the next episode, we have a $60,000 copy paste mistake. So listen to that one.

Scott Tolinski

That's a, yeah, a great one for the headline there. 60,000 copy paste mistake. I used to have to, when I worked at a record label, I used to have to wire royalties out all the time, and that was by far the most stressful part of the job. You're you're you're walking out of the bank. You're filling out that piece of paper and having to do it, like, once a week. I remember the 1st time I did it. I felt like, oh, I cannot believe this company is putting me in charge of sending out their royalties like this.

Scott Tolinski

Alright. Last one here. I was once working for an agency on a high profile customer.

Scott Tolinski

We were preparing for one of their peak days and needed To make some changes to their payment gateway integration so they could offer better service to their customers.

Topic 13 26:50

Forgot to remove test payment integration code

Scott Tolinski

Working on the payment integration is Always scary. Very. But, yeah, as we have just talked about.

Scott Tolinski

But all went fine and worked as expected.

Scott Tolinski

We let it rest and waited a few days before shipping it to the website. Next morning, we deployed and started working on other features.

Scott Tolinski

Within an hour, we suddenly got a call from their CMO who told us that customer service was getting calls from a customer Who couldn't pay? We sprang into action and looked into the changes I had shipped.

Scott Tolinski

I made a hard coded switch to make sure I was Testing with the gateway.

Scott Tolinski

Oh, this is this is classic. Oh my gosh.

Scott Tolinski

I had made a hard coded switch To make sure I was testing with the test gateway, which I had forgotten to remove again. OMG.

Scott Tolinski

This rejected all payment from the production site.

Scott Tolinski

Fortunately, we could make the correction and deploy it, but this was back when deployments could easily take half an hour. Oh, yeah. So by the time it was out, they had lost several 100 orders.

Scott Tolinski

Nothing to do but own up to my mistake and make sure it wouldn't happen again.

Scott Tolinski

Oh. Man. Hey. Guess who's done this? I'm raising my hand right now. I used to, you know, level up tutorials. You're managing those the sandbox environments. You're managing the production environments. You're testing your payments. Next thing you know, you know, there there's some sort of bug. You gotta get it up and fixed. Next thing you know, that Production key and, sandbox key gets swapped somehow in your e n v, and you're you're pooched. Yeah. Because

Wes Bos

I don't I've I don't think I've ever done that. I've almost accidentally done it. I I think we had this last year. Somebody submitted a a thing where they deployed it. And if you put real credit cards into a test environment, They'll work, you know, but they won't actually charge your credit card anything. Depends on who the provider is.

Scott Tolinski

Brain brain the way Braintree works, you actually have to have a specific type of credit to get a certain response. Oh, man. He's trying to the provider is. Yeah. Yeah.

Scott Tolinski

It certainly depends on who the provider is. I know there's some of them that do a range, so there's a certainly a chance it could work. My situation, Wes, was I had a Staging site that had used the sandbox. Oh, yeah. The staging site was an entirely different code base. And when I was ready to swap over, I was like, oh, yeah. I'll just make the staging site live. Oh, shoot. I forgot about the most important part, which was v and v variable change. Oh, I did it once Where

Wes Bos

when I before I release a course, I usually let, like, 100 people buy it and and go through it and whatnot. So I had released it to the like a 100 people that wanted, like, early access and they all got emails with, like, a local domain in the email, and they weren't but they had paid real money for it, but they weren't able to, like, activate it Because the URLs were whatever. And luckily, thank goodness, they're all web developers, so I was able to say, hey.

Wes Bos

Replace local hosts with this URL and you're good to go. But imagine that times Not having smart customers

Scott Tolinski

and Yeah.

Wes Bos

Like, times a 1000000 customers.

Wes Bos

Yeah. Nightmare. So if If you enjoyed this, hopefully you're not too stressed out. We got another one coming for you on Wednesday. Some doozies doozies in there, so make sure you tune in to that one as well. Please submit your own. We'll we'll put them on the show for next year.

Wes Bos

Email me at wes@westboss.com.

Wes Bos

Email Scott. Tweet us. Whatever you gotta do, We love hearing them. Yeah. And in fact,

Scott Tolinski

I have a new email, scott@syntax.fm, so you can hit me up at a syntax email. That's pretty sweet. I I do wanna say, just like you said, Wes, at the jump of this episode, this is just a taster for what you're about to get on Wednesday, because What you will get on Wednesday is a barrage of of some of the spookiest stories we've ever had on this podcast, including, one of them that cost, Well The 1st $1,000,000 mistake 1st $1,000,000 mistake is syntax, spooky story history. So if that doesn't get you, hyped up, I don't know what will because that

Wes Bos

that one to me had me rolling. It it, yeah. No. It gets really bad. $1,000,000 in, like, potential revenue. Like, literally He lost $1,000,000.

Scott Tolinski

Alright. Well, buckle up, folks. It's about to get really spooky on Wednesday. So Thanks so much for listening, and we'll catch you on Wednesday. Peace. Peace.

Scott Tolinski

Head on over to syntax dotfm for a full archive of all of our shows.

Scott Tolinski

And don't forget to subscribe in your podcast player or drop a review if you like this show.

Wes Bos

My name is Wes. My dog eats food on

Scott Tolinski

the moon.

Share