","Status__c":"Resolved","Start_Date__c":"2021-12-15","Last_Updated__c":"2021-12-15T16:42:16.000+0000","LastModifiedDate":"2021-12-15T16:42:16.000+0000","Category__c":"Service Disruption","Is_Mis_Red__c":false,"Size__c":"Large","Impacted_Cells__c":"okta.com:5;okta.com:7;okta.com:12;okta.com:14","Impacted_Audience__c":"Admin","Service_Feature__c":"Core Service","Okta_Sub_Service__c":"Core Platform","Start_Time__c":"2021-12-15T15:20:00.000+0000","CreatedDate":"2021-12-15T15:54:07.000+0000","CurrencyIsoCode":"USD"},{"attributes":{"type":"Incident__c","url":"/services/data/v57.0/sobjects/Incident__c/a9C4z000000wk5IEAQ"},"Id":"a9C4z000000wk5IEAQ","Duration__c":0,"Log__c":"Okta is currently experiencing an issue with third-party SMS provider issue with ATT service. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P1Y000005V3RCUA0"},"Id":"a1P1Y000005V3RCUA0","CreatedDate":"2018-12-13T14:51:42.000+0000","IncidentId__c":"a9C1Y000000GpJuUAK","UpdateLog__c":"Resolved: The issue impacting CORS authentication for customers using OIDC applications and API AM in Okta Preview Cells has been fixed. The issue could prevent Helpdesk Admins from being able to reset a users password via a temporary password. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P1Y000005V49YUAS"},"Id":"a1P1Y000005V49YUAS","CreatedDate":"2019-02-19T10:25:14.000+0000","IncidentId__c":"a9C1Y000000GpKxUAK","UpdateLog__c":"On Feb 7th at approximately 4:30PM PDT, Okta began receiving reports from customers in US Cell 4 who were experiencing errors while attempting to provision users to Office 365 or while modifying Office 365 configuration settings. US Cell-11 is recovering. We are still actively investigating these issues to isolate a fix. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P1Y000009QRjCUAW"},"Id":"a1P1Y000009QRjCUAW","CreatedDate":"2021-02-05T23:12:18.000+0000","IncidentId__c":"a9C1Y0000008PgbUAE","UpdateLog__c":"Root Cause Analysis:\r\nOn Tuesday February 2nd at approximately 3:46 pm (PST) Okta began receiving customer reports that end users were unable to authenticate into their Microsoft Office 2012 Desktop Clients applications on Microsoft Windows.\r\n\r\nUpon investigation, Okta determined that this failure was caused by an update to the Okta Sign In Widget which had already been deployed to all Preview and Production cells except US Cell 5. The intermittent issues were reported for US Cells OK 3, 6, 7, 11, 12, and 14, but the issue was not cell specific since the root cause was outside of the Okta service. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P1Y000007zK4ZUAU"},"Id":"a1P1Y000007zK4ZUAU","CreatedDate":"2020-01-23T01:16:26.000+0000","IncidentId__c":"a9C1Y0000004E8uUAE","UpdateLog__c":"Okta has resolved an issue in US Cell 1 impacting AD/LDAP Agent state reporting. We will provide an update in 30 minutes or sooner if additional information becomes available. Clearing browser cookies will help mitigate the Multifactor Authentication redirect loop. The SMS service as the 2FA method worked as designed. Our application team doesn't recommend reconfiguring Microsoft O365 Federation. During the incident, some Okta AD agents were unable (in some cases intermittently) to make outbound connections to the Internet and thereby, were unable to connect to the Okta service. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z000009mReFEAU"},"Id":"a1P4z000009mReFEAU","CreatedDate":"2022-11-07T16:17:18.000+0000","IncidentId__c":"a9C4z000000YzhlEAC","UpdateLog__c":"We are experiencing a Trust Event that is impacting email delivery with multiple domain providers at this time, which can impact both CIAM and Workforce customers.\r\nFor CIAM, the impacted domains include: outlook.com, hotmail.com (hotmail.co.uk, msn.com, hotmail.fr, etc. Okta has since concluded that hackers used this malicious access to copy code repositories associated with Workforce Identity Cloud (WIC), the organizations enterprise-facing security solution. This bug caused a failure to successfully provision users and save Office 365 app settings. Okta also re-enabled the API for adding groups to projects, fully restoring all services at this time.\r\n\r\nOkta's has implemented changes to mitigate the impact of similar issues in the future, and is continuing its full Root Cause Analysis of the incident,. Our monitoring shows recovery and the next update will be in 3 hours or sooner if additional information becomes available. Jobs that were terminated would get triggered again on the next scheduled import. Engineering is working to identify the list of customers still impacted and is investing a way to proactively address the issue for these customers. Outlined below are the facts regarding this incident. Once recovered, the front pages were restored to both cells. This fix was submitted to the app store and expedited as Okta Verify 5.0.7 at 5:20pm PDT on April 16th, the build was approved and published at 9:20am PDT on April 17th.\r\n\r\nOkta is expanding migration testing scenarios across more versions to prevent future occurrences as well as incorporating additional monitoring to detect similar issues more quickly in the future. In the event that you need assistance in implementing the resolution guide, please contact customer support for assistance. The next update will take place at 12:45pm PT or as soon as additional details become available. Another update is planned for 6:25pm PT or sooner as additional information becomes available.
","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P2A00000FgbiDUAR"},"Id":"a1P2A00000FgbiDUAR","CreatedDate":"2018-03-21T01:27:26.000+0000","IncidentId__c":"a9C2A0000004CSnUAM","UpdateLog__c":"Okta continues to investigate an issue with Provisioning Setup and IDP-Initiated Single Sign-On functionality for the following Office 365 Apps: Yammer, CRM and Teams. At 10:50AM Oktas infrastructure provider began to automatically limit the volume of traffic to US Cell 6 to mitigate impact to downstream services. This caused these worker servers to be taken out of service, which propagated the impact and caused unavailability of Workflows Console and Event Hook processors.\r\n\r\nRemediation Steps\r\n\r\nOkta took action to identify the affected platform services, and restarted these services to restore them to health adding additional capacity to aid in faster recovery. Recently an upgrade was introduced to expire existing device token cookies beginning November 1 gradually over a 28 day period, in order to implement a new cookie type.\r\n\r\n \r\n\r\nIt was determined after investigation that there is another function within Okta which incorrectly generated the device token cookie again if it expired leading into the flow, and there was insufficient test validation for the operation involving cookie logic to catch this failure. The issue has been resolved as of 2:16 PM Pacific time. We are committed to implementing improvements to the service to prevent future occurrences of this incident.\r\n\r\nDetection and Impact: \r\n\r\nOn October 19 at 6:24PM PT, Okta began receiving alerts on Cell 3 and started to investigate. Say Goodbye to Charging Stops. Additional root cause information will be provided by Wednesday 2/23. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P2A00000G0Uq2UAF"},"Id":"a1P2A00000G0Uq2UAF","CreatedDate":"2018-03-07T21:50:15.000+0000","IncidentId__c":"a9C2A000000PBikUAG","UpdateLog__c":"An issue impacting API Access Management for all end-users in US Preview Cell was resolved. For impacted customers that provisioned Workday users to Active Directory (AD), where group members were part of delegated authentication to AD, affected users were unable to log into Okta.\r\n\r\nAt approximately 1:10pm PST, Okta took action to prevent further customer impact by disabling user import jobs on US Cell 3. The failover mechanism has mitigated the impact. If you need immediate assistance in resolving the issue, please contact Okta Customer Support. Following resumption of normal operation, Okta successfully added a router with correct certificates back into the router group for US Cell12.\r\n\r\n\r\nPreventive Actions\r\n\r\nOkta is improving the process for deploying and verifying SSL certificates as well as adding detection measures to prevent occurence of this issue in the future. We are currently deploying a fix across all cells. We encourage you to go to the latest updates on Microsoft's Status at [status.okta.com](http://status.office.com). Our next update will be in 12 hours or sooner if additional information becomes available","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z000007sujAEAQ"},"Id":"a1P4z000007sujAEAQ","CreatedDate":"2022-07-28T19:45:35.000+0000","IncidentId__c":"a9C4z0000012OTZEA2","UpdateLog__c":"An issue impacting network connectivity in US Cell-6, US Cell-10 and US Cell-11 was addressed. End users enrolled in SMS MFA with a Google Voice number may not receive SMS MFA codes. At 5:23 PM PDT Okta engineering confirmed the fix had been deployed and validated across all impacted cells.\r\n\r\nOkta is implementing additional testing to protect against future regressions related to changes in role-based permission settings. This resulted in request processing time to increase significantly to the point that auto scaled services were not sufficient to mitigate the impact to the increasing number of requests. We have confirmed that all services have been restored. \r\n\r\nOkta would like to thank customers and partners for their collaboration and partnership in reporting and investigating issues that arise in the pre-production sandbox environments before they might manifest in production cells. SP-Initiated Single-Sign-On is not affected. This was due to an incorrect handling of legacy encryption keys that were used in previous Okta Verify versions.\r\n\r\nBecause of the complexity of the issue, Okta expedited a temporary build that would rollback the functionality for users affected as the full fix was worked on to support all upgrade paths. Okta has determined there is no meaningful impact to Okta Verify at this point. We will provide our next update at 9 AM PST, or earlier if deemed necessary. ITWeb Security Summit will feature over 50 speakers, covering today's hottest cyber security topics, from cyber warfare to DevSecOps, cloud security, incident response, cyber security psychology . Users attempting to authenticate into RingCentral via the Okta service were not impacted, however, user administrators attempting to provision or deprovision users within the RingCentral application would have been unable to complete profile updates during this time.\r\n\r\nAt 5:30 PM PST on May 26th Okta reverted to the prior release of the affected service and RingCentral provisioning operations were observed to complete as expected. The status of US Cell-11 is being monitored. We are committed to implementing improvements to the service to reduce the likelihood of any future occurrences of this kind.\r\n\r\n \r\nDetection and Impact: \r\n\r\nOn November 4th at 1:00 pm (PT), Okta was alerted to email deliverability issues related to third party email service providers. \r\n\r\nAt approximately 1:49 PM PDT Okta verified a fix and began deploying the change as a patch to the impacted cells. Additional root cause information will be provided here as soon as it is available. End users enrolled in SMS MFA with a Google Voice number may not receive SMS MFA codes. The issue was fully resolved on Dec 13th by 6:26am PST in US Preview Cell 1 and by 6:38am PST in EU Preview Cell 2. We are committed to implementing additional improvements to the service to prevent future occurrences of this incident.\r\n\r\nTiming:\r\nOn August 6th at 5:27AM PDT, customers began reporting intermittent delays for SMS MFA to US based numbers. Our next update will be in the next 30 min. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z00000BAKm4EAH"},"Id":"a1P4z00000BAKm4EAH","CreatedDate":"2021-10-27T20:40:13.000+0000","IncidentId__c":"a9C4z000000wk5IEAQ","UpdateLog__c":"Our monitoring shows a return to normal conditions with MFA for customers that utilize SMS with AT&T. Okta continues to investigate a solution for users running versions of Android below 6.0 as well as those relying on downloading the Okta Mobile APK from their Okta tenant. Clearing browser cookies will help mitigate the Multifactor Authentication redirect loop. However, because of the issue, Cell 5 experienced a spike in access requests (429s) which resulted in some users encountering access violations until 11:07AM(PT), at which time traffic access returned to normal rates. Additional update targeted for 3:00pm PDT. As Okta maintains a restrictive firewall policy that defaults to denying access unless expressly allowed by a rule, rules revocation further restricted inbound traffic to hosts and caused traffic to be blocked. The service interruption was fully restored at 6:31AM PST.\r\nAdditional root cause information will be available within 2 Business days. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P1Y000005qK3nUAE"},"Id":"a1P1Y000005qK3nUAE","CreatedDate":"2019-09-05T19:56:49.000+0000","IncidentId__c":"a9C1Y000000fxj2UAA","UpdateLog__c":"Okta has identified the cause of the Activation Email regression affecting new Developer tenants and is currently preparing to deploy a fix. Admins may continue to experience delays in job processing while we investigate the root cause. At Okta, trust and transparency are our top priorities. See below. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z00000BAKsCEAX"},"Id":"a1P4z00000BAKsCEAX","CreatedDate":"2022-02-02T20:16:14.000+0000","IncidentId__c":"a9C4z000000wkADEAY","UpdateLog__c":"Endusers receive JS error when logging into the application when using embedded browser. We are working with the service provider to investigate the root cause of this service disruption and will post the RCA to [status.okta.com](http://status.okta.com.com) within 48 hours. We will provide our next update at 12 PM PST or earlier if deemed necessary","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z00000BAKnMEAX"},"Id":"a1P4z00000BAKnMEAX","CreatedDate":"2021-11-11T01:47:05.000+0000","IncidentId__c":"a9C4z000000wk5cEAA","UpdateLog__c":"Root Cause Analysis\r\n\r\nWe sincerely apologize for any impact this incident has caused you, your business or your customers. At 6:30 AM PST, our monitoring shows improvements in network connectivity. Some users cannot verify their identity by acknowledging the push notification sent to their iPhone. \r\n\r\nRoot Cause Summary:\r\n\r\nA script with erroneous DNS configurations was executed at 6:22 PM PT in Cell 3, and made changes to DNS records for *.okta.com domains.\r\n\r\nRemediation Steps:\r\n\r\nAfter confirming the details of the incident, Okta successfully remediated the issue by resetting the impacted DNS changes at 6:42PM PT. Users attempting to authenticate with Office 365 are receiving a \"Throttled\" message. The end users will be able to download version 7.9.2 as the Google Play store makes it available. Additional root cause information will be available within 2 Business days. Okta strongly recommends that customers always configure multiple MFA factors, so that in cases such as this, where the telecommunications infrastructure is impacted, end users are able to authenticate leveraging another factor. During this time, customers in these cells received 5XX and Proxy Error: Error during SSL Handshake with remote server errors when attempting to access their login page. Additionally, Okta plans to implement a connection-pooling proxy in front of the primary ASA database cluster to prevent future occurrences of this issue. We will provide an update in 30 minutes or sooner if additional information becomes available. ","CurrencyIsoCode":"USD"},{"attributes":{"type":"IncidentUpdate__c","url":"/services/data/v57.0/sobjects/IncidentUpdate__c/a1P4z000009mS5aEAE"},"Id":"a1P4z000009mS5aEAE","CreatedDate":"2023-01-23T18:53:14.000+0000","IncidentId__c":"a9C4z000000YzkkEAC","UpdateLog__c":"The issue impacting specific Microsoft O365 (Word, Excel, PowerPoint) IdP-initiated login for all end users has been resolved. Several customers have publicly chastised Okta for a slow drip of information that left them uncertain about what to do. We are committed to implementing improvements to the service to prevent future occurrences of this incident.\r\n\r\nDetection and Impact \r\n\r\nOn May 11 at 6:00 am PDT, Okta Technical Operations teams were alerted to an issue where some Okta users experienced an increase in error rates, slow response times and may have received HTTP 429 Too many requests response code in US-Cell 2, due to an issue with one of our databases.