This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
You know what's worse than not being able to log in?
Being told everything worked right up until you try to actually use your account.
That was the bug.
And, somehow, once again, I ended up being pulled into my favourite corner of software engineering: authentication.
At this point, I'm beginning to suspect authentication bugs have a personal grudge against me. 😅 In my previous Smash Story, I wrote about a bug where users simply couldn't log in.
That is obviously bad, especially when authentication is the gate standing between a user and everything they are trying to access.
This time, though, the bug was sneakier.
The signup flow worked, the account was approved, the background task ran, the email and SMS were sent, and the user received the PIN they were expecting.
Everything looked perfectly healthy until they actually tried to use the account, at which point the PIN didn't match what Cognito expected and the confirmation journey fell apart.
In other words, the system was confidently telling the user, "Yep, we're all good!" while quietly making sure they couldn't actually get through the door.
And that turned out to be a much more interesting bug to debug.
It Started With a Fairly Complicated Authentication Setup The application had a serverless .NET backend running on AWS Lambda, a SQL Server database, and an Angular frontend communicating with the backend through API Gateway.
AWS Cognito handled authentication, while SES and SNS took care of transactional emails, SMS messages and verification codes.
The authentication architecture had also evolved over time, which meant we weren't dealing with one beautifully organised User Pool where every authentication requirement had been designed together from day one.
There were actually two Cognito User Pools, each with a different job.
The newer pool supported the newer authentication experience and was involved in sending the PIN to users, while the legacy pool was still responsible for older authentication flows, including signup using a mobile number.
Alongside those pools, the application database had its own table representing the same people.
So a single user could effectively be represented in several places, with the database holding the application's current user information, Legacy Cognito holding the authentication record used by the older flow, and the newer authentication flow handling the PIN and account confirmation experience.
None of that was inherently wrong.
The problem was that these systems had to agree about the same person, and one of the identifiers they relied on could change.
As long as everything stayed aligned, nobody noticed.
The moment one representation moved ahead of another, however, things got interesting very quickly.
The Flow That Looked Completely Fine The problem appeared in the partner and dependant journey.
A member could create a partner or dependant either during signup or later from the member details area.
A relevant non-member user would then approve the partner or dependant, but that approval didn't immediately create the Cognito user and send all the onboarding information.
Instead, it scheduled an asynchronous task called in a table.
That task ran every 15 minutes, which meant there was a very real gap between "this person has been approved" and "let's go and create everything they need." Normally, that gap didn't matter.
But imagine a partner gets approved at 10:01 AM and the background task is sitting there waiting for its next run.
At 10:05 AM, while the task is still minding its own business, the partner changes their name.
That sounds completely harmless, because, well, it is completely normal for someone to change their name.
Except the username for partner and dependant accounts was derived from their details, so changing their first or last name could also change the username represented in the database.
The person was still the same person.
Their email address was still the same.
Their approval was still valid.
But the database now had a different username from the one that had existed when the approval was originally processed.
And that five-minute gap was about to become very important.
Then the Background Task Woke Up Eventually, the 15-minute task ran.
It looked at the latest information in the database and attempted to create the Cognito user using the latest username.
That sounds like exactly what a well-behaved background task should do, right?
Unfortunately, Legacy Cognito had already seen this email address before.
There was an existing Cognito user associated with the same email, but it was using the old username.
Cognito therefore rejected the new user creation because the email address was already associated with an existing user.
And here is where the fun really began.
The database had the latest username, while Legacy Cognito still had the old one.
The application had tried to create the user it thought it needed, but Cognito couldn't create it because, from Cognito's perspective, that email address was already taken.
To make matters worse, Cognito usernames are immutable, so we couldn't simply take the existing user and change its username to the latest value.
We had the same human being represented in both systems, but the systems disagreed about which username represented that person.
The person hadn't changed.
The data had.
And authentication really doesn't enjoy that distinction.
Then the User Got Their PIN This was the part that made the bug particularly deceptive.
The user still received the expected email and SMS.
There wasn't some dramatic failure message announcing that the account creation had gone sideways.
The background task had run, the notifications had been sent, and there was a Cognito user associated with the email address, so from the outside, the whole process looked remarkably healthy.
Naturally, the user did exactly what we had asked them to do: they opened the confirmation flow, entered the PIN they had just received and tried to finish setting up their account.
Except the PIN being sent to the partner or dependant didn't correspond to what was stored against the Cognito user the application expected.
So the user couldn't confirm the account.
And without successfully completing that step, they couldn't get through the rest of the authentication journey either.
This was the sneaky bit: the system hadn't obviously failed when the original problem occurred.
The actual inconsistency had been introduced when the user changed their name after approval but before the asynchronous task ran.
The user only discovered the consequences several steps later, when they were standing at the confirmation screen wondering why the perfectly legitimate PIN they had just received suddenly wasn't legitimate enough for the system.
From the user's perspective, it was a completely reasonable question. "You sent me this code.
Why are you telling me it's wrong?" From the system's perspective, unfortunately, there was a very long answer.
So We Followed the User Backwards The obvious place to start was authentication.
When someone can't confirm their account or sign in, you naturally start looking at the PIN, passwords, Cognito responses, frontend requests and anything else happening around the login flow.
But the further we investigated, the clearer it became that authentication wasn't actually where the problem had started.
It was simply where the problem had decided to make itself visible.
So we followed the journey backwards.
We went from the failed confirmation, back to the PIN, back to the background task, back to the approval process and eventually back to the database and Cognito records.
That was the moment everything clicked.
The database had the latest username.
Legacy Cognito had the old username.
The email address was the same.
The asynchronous task had tried to create a user with the new username.
Cognito had rejected that creation because th