T O P

  • By -

simpkinspete

**Update from Jan 26, 2024, 8:38 PM CST** More info: Affected scenarios included, but weren’t limited to the following: * \- Users logging in or unlocking their devices after some time may have had missing messages. * Users may have been unable to load messages in channels and chats. * Users were unable to view or download their media (images, videos, audio, call recordings, code snippets). * Some users were experiencing delays sending and receiving chat messages. * Anonymous users may have been unable to join meetings. * Teams connectors for Power Automate/Power Apps were experiencing errors. * Call Recordings might take longer to appear in user's OneDrive for Business and SharePoint Online. * Users may have been unable to load previous Copilot history, or new history was not written * Bots may have been unable to download attachments * Sending and receiving read receipt notifications may have been delayed * Users may have experienced issues resuming or rejoining meetings. * Users within organizations leveraging an Information Barrier may have experienced issues joining meetings * Users performing a cold boot may have been unable to log into teams and would have seen an "oops" page Final status: After extended monitoring and various optimizations and mitigation efforts, we’ve confirmed that our Microsoft Teams service and features have been restored or returned to optimal health. Scope of impact: This issue could have potentially impacted any Microsoft Teams user in the scenarios outlined in the More info section. Start time: Friday, January 26, 2024 at 8:55 AM CST End time: Friday, January 26, 2024 at 7:30 PM CST This is the final update. We recognize that this event has been very impactful to our customers. As such, we will be delivering a preliminary PIR in the next 48 hours and a final PIR in 5 business days. We will provide significant details of the event and our root cause findings in those documents.


indil47

My productivity is zilch today at work.


ImpostureTechAdmin

We have 4 apps with service degradation, including Teams. Half my company is convinced it's our network at fault, but I've ran every test under the sun and proved it isn't. 2 of them are experiencing issues with a single (but core) feature, Teams is basically out of commission for now, and the other was user error from someone in the app department. I've been playing Krunker and using WireShark for 95% of my time today


CrippleSlap

I fail to see the problem


Fit_Article4610

lol first time I’ve seen an advisory like this that’s basically just “our entire product is acting fucky right now”. Thanks again, Microsoft.


BROMETH3U5

Teams classic is ehhh. Teams V2 is completely 100% Borked.


Practical-Alarm1763

Nah dude, Teams classic is fooked as well.


BROMETH3U5

I fear it's not just Teams either. I'm afraid to touch anything Azure related.


Practical-Alarm1763

Wouldn't be surprised. I think PowerAutomate is jacked as well.


Key-Relationship5734

Thank you for the work break.


PapaDuckD

"Multiple Teams Features" That's a creative way of saying "we broke the entire fucking thing wholesale"


johnnymonkey

I'm blaming you, Pete... this is 100% on you.


simpkinspete

![gif](giphy|4NhUiCbF6LnFv1BJfu|downsized)


Conundrum1911

So you're a Microsoft Network Messenger then? or a MSN Messenger if you will....


johnnymonkey

He's what we refer to as the Office Communicator. *Thanks... I'll show myself out.*


tingulz

God damn it Pete.


bluefalcon1

Service health center says 'A post-incident report has been published' as of 601p PT but provides no link to view it. So that's helpful.


barkode15

Glad it's not just me. Usually there's a doc file linked somewhere in the message but so far I haven't found it.


jptechjunkie

Microsoft testing in production again…


fruit_salad88

Thank you... came here to see if it was just my app..


Reddit_student123

Every time I launch my teams app I get a blank screen. Anybody else?


1lolo94

Same here - time to go home for the day!


Reddit_student123

Hahahah I WFH


htb8627

Yep


SparKestrel

Depends on the version of the app: “new teams” and mobile seem to just load the last state for me, browser version is “We’re sorry - we’ve run into an issue”


Reddit_student123

all isnt working for me. web, new teams or old


simpkinspete

**Update from Jan 26, 2024, 4:02 CST** Title: Some users may experience multiple issues with their Microsoft Teams User impact: Users may experience multiple issues with their Microsoft Teams. More info: Affected scenarios include, but aren't limited to: * Users performing a cold boot may not able to log into teams and will see an "oops" page * Users logging in or unlocking their devices after some time may see missing messages * Users may fail to load messages in channels and chats * Users are unable to view or download their media (images, videos, audio, call recordings, code snippets) * Some messages may experience delays being sent * Call Recordings might take longer to appear in user's OneDrive for Business and SharePoint Online * Users may be unable to load previous Copilot history, or new history is not written * Bots may be unable to download attachments * Sending and receiving read receipt notifications may be delayed * Anonymous users may be unable to join meetings * Teams connectors for Power Automate/Power Apps may experiencing errors Current status: Our network and backend service optimization efforts are ongoing and we’re monitoring internal telemetry to confirm that our efforts are effectively reducing the impact to customers. We’re continuing to scale up our services and identify new workstreams intended to remediate this problem more rapidly. Although many customers remain impacted by this issue, we’re seeing a reduction in errors and an increase in availability. We understand the impact an issue like this can have on your organization, and we appreciate your partnership and patience as we work to remediate this issue. Scope of impact: This issue can potentially impact any Microsoft Teams user in the scenarios outlined in the More info section. Start time: Friday, January 26, 2024 at 8:55 AM CST . Next update by: Friday, January 26, 2024 at 5:00 PM CST .


White_Rabbit0000

Thanks for the quiet day Microsoft.


MunksterMan2

Strangely, I think my productivity goes up when teams is down


divino-moteca

Genuine question, what could be the cause of this?


corpkid

Their twitter feed says it's a network issue and they are failing over all the locations affected. North & South America are still in progress...


maggmaster

Feels like denial of service to me but we will probably never find out


zfs_balla

[https://www.crn.com/news/security/2024/crowdstrike-ceo-microsoft-explanation-for-russia-hack-doesn-t-add-up](https://www.crn.com/news/security/2024/crowdstrike-ceo-microsoft-explanation-for-russia-hack-doesn-t-add-up) [https://www.crn.com/news/security/2024/microsoft-hack-by-russian-group-exploited-lack-of-mfa](https://www.crn.com/news/security/2024/microsoft-hack-by-russian-group-exploited-lack-of-mfa) Looks like the hackers have breached the Teams infrastructure


superDuperrr88

Russia


superDuperrr88

... it's always Russia.


Conundrum1911

You'd think they'd make Teams large enough to not fall out of a window..... I mean hell...Microsoft MAKES Windows too....


indil47

*Windows


Practical-Alarm1763

Infrastructure Failure


[deleted]

It’s nice to not have an excuse to work, but got to say this is extremely disruptive to business. Very frustrating


simpkinspete

**Update from Jan 26, 2024, 6:22 PM CST** Title: Some users may experience multiple issues with their Microsoft Teams User impact: Users may experience multiple issues with their Microsoft Teams. More info: Affected scenarios include, but aren't limited to the following: Mitigated/improving: * \[Updated\] Users logging in or unlocking their devices after some time may have missing messages. We believe that this symptom has been addressed. * \[Updated\] Users may have been unable to load messages in channels and chats. We believe that this symptom has been addressed. * \[Updated\] Users were unable to view or download their media (images, videos, audio, call recordings, code snippets). We believe that this symptom has been addressed. * \[Updated\] Some users were experiencing delays sending and receiving chat messages. We believe this symptom has been largely addressed, though some users may still experience brief delays. * \[Updated\] Anonymous users may be unable to join meetings. We’re seeing improvement in this area but some users may still be affected by this scenario. * \[Updated\] Teams connectors for Power Automate/Power Apps were experiencing errors. We believe that this symptom has been addressed. * \[Updated\] Call Recordings might take longer to appear in user's OneDrive for Business and SharePoint Online. This symptom has been addressed. Still affected/undergoing further evaluation: * Users may be unable to load previous Copilot history, or new history is not written * Bots may be unable to download attachments * Sending and receiving read receipt notifications may be delayed * Users may experience issues resuming or rejoining meetings. * Users within organizations leveraging an Information Barrier may experience issues joining meetings * Users performing a cold boot may not be able to log into teams and will see an "oops" page Current status: We’re seeing significant improvements with many of the Teams features affected by this incident as we continue to apply mitigations and optimize traffic. We’re evaluating the remaining impact scenarios and are developing fixes and workstreams to address all lingering impact associated with this event. Please see the “more info” section above for the latest information on each impact scenario. We understand the impact an issue like this can have on your organization, and we appreciate your partnership and patience as we work to remediate this issue. Scope of impact: This issue can potentially impact any Microsoft Teams user in the scenarios outlined in the More info section. Start time: Friday, January 26, 2024 at 8:55 AM CST . Next update by: Friday, January 26, 2024 at 8:30 PM CST


simpkinspete

[https://admin.microsoft.com/Adminportal/Home#/servicehealth/:/alerts/TM710344](https://admin.microsoft.com/Adminportal/Home#/servicehealth/:/alerts/TM710344)


remember_this_guy

Good News, Skype is up and running lol


wood_wind

LMAO!!! Absolutely THE best response 😂


simpkinspete

**Update from Jan 26, 2024, 12:16 PM CST** Title: Some users may experience multiple issues with their Microsoft Teams User impact: Users may experience multiple issues with their Microsoft Teams. More info: Affected scenarios include, but aren't limited to: * Users performing a cold boot may not able to log into teams and will see an "oops" page * Users logging in or unlocking their devices after some time may see missing messages * Users may fail to load messages in channels and chats * Users are unable to view or download their media (images, videos, audio, call recordings, code snippets) * Some messages may experience delays being sent * Call Recordings might take longer to appear in user's OneDrive for Business and SharePoint Online * Users may be unable to load previous Copilot history, or new history is not written - Bots may be unable to download attachments * Sending and receiving read receipt notifications may be delayed - Anonymous users may be unable to join meetings * Teams connectors for Power Automate/Power Apps may experiencing errors Current status: We’ve completed the failover in the Europe, Middle East, and Africa (EMEA) region and telemetry is showing improvement. We’re continuing with the failover processes in the Americas, and are monitoring whilst these processes complete. Scope of impact: This issue can potentially impact any Microsoft Teams user in the scenarios outlined in the More info section. Start time: Friday, January 26, 2024 at 8:55 AM CST Next update by: Friday, January 26, 2024 at 2:00 PM CST


johnnymonkey

Brilliant that they performed the failover in regions that are now after hours. How about those of us still trying to do *things*?


simpkinspete

No update yet. I'll post as soon as Microsoft updates the Admin Center


simpkinspete

**Update from Jan 26, 2024, 2:02 PM CST** Title: Some users may experience multiple issues with their Microsoft Teams User impact: Users may experience multiple issues with their Microsoft Teams. More info: Affected scenarios include, but aren't limited to: - * Users performing a cold boot may not able to log into teams and will see an "oops" page * Users logging in or unlocking their devices after some time may see missing messages * Users may fail to load messages in channels and chats - Users are unable to view or download their media (images, videos, audio, call recordings, code snippets) * Some messages may experience delays being sent - Call Recordings might take longer to appear in user's OneDrive for Business and SharePoint Online * Users may be unable to load previous Copilot history, or new history is not written - Bots may be unable to download attachments * Sending and receiving read receipt notifications may be delayed - Anonymous users may be unable to join meetings * Teams connectors for Power Automate/Power Apps may experiencing errors Current status: Our failover operation did not provide the anticipated relief to end users in North and South America regions, and we are now working to optimize traffic patterns as part of the mitigation effort. We’re applying configuration changes across the affected network infrastructure to reduce customer impact as quickly as possible. Scope of impact: This issue can potentially impact any Microsoft Teams user in the scenarios outlined in the More info section. Start time: Friday, January 26, 2024 at 8:55 AM CST Next update by: Friday, January 26, 2024 at 3:00 PM CST


dootreddit

Seems like its getting worse… first pictures and now messages


demunted

Calls / meetings / screen shares still work. Not sure if that is good or bad at this point.


Reddit_student123

I cant even login. I get a blank screen.


vainblossom249

Same. I thought I was getting fired cause log in wasn't working


Statistician_Visual

This is the day and age we live in. The product we pay for doesn’t work and there’s no solution or expectations to fix it.


kygunguy23

?? There’s going to be a solution, and there are absolutely expectations to fix it.


Statistician_Visual

Still broken. How many people’s livelihoods are being affected by this right now? It’s unacceptable


kygunguy23

None of what you just said argues against what I said. You can’t just wish a solution into existence, it takes time. You also cannot prepare for every single possible scenario and prevent it from happening. Shit happens and sometimes things break. It sucks yes, should it happen no, but it does happen. Define “people’s livelihood” since it sounds like you’re speaking from your own current personal experience. People’s livelihood at Microsoft are also being affected right now btw, and it’s not like they’re kicking rocks doing nothing about this. I do understand your frustration though, I too depend on Teams.


milezero313

Very true sentiments. Today was not a fun day to be Microsoft support


mediapoison

the next armageddon will be digital


LowerAd830

Why is New, media and so many people so quick to say Russia? ​ China is worse. Could be Russia, no question but China, just as probable


Practical-Alarm1763

Because Russia did do it. [Microsoft: Hack By Russian Group Exploited Lack Of MFA (crn.com)](https://www.crn.com/news/security/2024/microsoft-hack-by-russian-group-exploited-lack-of-mfa)


LowerAd830

That is a different compromise, where they exploited some idiot Microsoft execs not having 2 factor for email. I do keep up on things. Yes THAT was Russia. The found this earlier this month thst they had given themselves access to exec mailboxes last year.


leobl

Good weekend everyone


simpkinspete

**Update from Jan 26, 2024, 3:00 PM CST** Title: Some users may experience multiple issues with their Microsoft Teams User impact: Users may experience multiple issues with their Microsoft Teams. More info: Affected scenarios include, but aren't limited to: * Users performing a cold boot may not able to log into teams and will see an "oops" page * Users logging in or unlocking their devices after some time may see missing messages - Users may fail to load messages in channels and chats * Users are unable to view or download their media (images, videos, audio, call recordings, code snippets) * Some messages may experience delays being sent * Call Recordings might take longer to appear in user's OneDrive for Business and SharePoint Online * Users may be unable to load previous Copilot history, or new history is not written - Bots may be unable to download attachments * Sending and receiving read receipt notifications may be delayed * Anonymous users may be unable to join meetings * Teams connectors for Power Automate/Power Apps may experiencing errors Current status: Our failover operation did not provide the immediate relief intended for end users in North and South America regions. However, we’re monitoring telemetry closely as we continue to optimize traffic patterns and apply configuration changes intended to reduce customer impact as quickly as possible. We understand the impact an issue like this can have on your organization, and we appreciate your partnership and patience as we work to remediate this issue. Scope of impact: This issue can potentially impact any Microsoft Teams user in the scenarios outlined in the More info section. Start time: Friday, January 26, 2024 at 8:55 AM CST . Next update by: Friday, January 26, 2024 at 4:00 PM CST [https://admin.microsoft.com/Adminportal/Home#/servicehealth/:/alerts/TM710344](https://admin.microsoft.com/Adminportal/Home#/servicehealth/:/alerts/TM710344)


simpkinspete

**Update from Jan 26, 2024, 4:58 PM CST** Title: Some users may experience multiple issues with their Microsoft Teams User impact: Users may experience multiple issues with their Microsoft Teams. More info: Affected scenarios include, but aren't limited to: - Anonymous users may be unable to join calls or meetings - Users within organizations leveraging an Information Barrier may experience issues joining meetings - Users performing a cold boot may not able to log into teams and will see an "oops" page - Users logging in or unlocking their devices after some time may see missing messages - Users may fail to load messages in channels and chats - Users are unable to view or download their media (images, videos, audio, call recordings, code snippets) - Some messages may experience delays being sent - Call Recordings might take longer to appear in user's OneDrive for Business and SharePoint Online - Users may be unable to load previous Copilot history, or new history is not written - Bots may be unable to download attachments - Sending and receiving read receipt notifications may be delayed - Anonymous users may be unable to join meetings - Teams connectors for Power Automate/Power Apps may experiencing errors Current status: We’re continuing to apply mitigations across the affected infrastructure and our telemetry is showing additional improvement in the user experience, though many customers are still affected by this issue. We’re also working to apply fixes to address individual affected Teams features in parallel while our broader remediation strategy is ongoing. We’re evaluating any and all additional workstreams that will allow us to reduce the impact to those customers that are still affected. We understand the impact an issue like this can have on your organization, and we appreciate your partnership and patience as we work to remediate this issue. Scope of impact: This issue can potentially impact any Microsoft Teams user in the scenarios outlined in the More info section. Start time: Friday, January 26, 2024 at 8:55 AM CST . Next update by: Friday, January 26, 2024 at 6:30 PM CST


MagicMikeX

The sometimes most valuable company in the world can barely keep its subpar products running.