Is there an Movable Ink outage?
Movable Ink status: Systems Active
Last checked: 8 minutes ago
Get notified about any outages, downtime or incidents for Movable Ink and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 8 minutes ago
Get notified about any outages, downtime or incidents for Movable Ink and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Movable Ink.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| Da Vinci | Active |
| Link Redirect | Active |
| Recommendations | Active |
| Signal | Active |
| Web App | Active |
| Da Vinci - ESP Integrations | Active |
| Acoustic Campaign | Active |
| Cheetah Messaging | Active |
| Cordial | Active |
| Oracle Responsys | Active |
| SalesForce Marketing Cloud | Active |
| Studio | Active |
| Analytics | Active |
| Behavior & Conversion Scripts | Active |
| Content Rendering | Active |
| Customer Data API | Active |
| Data Sources | Active |
| Dynamic Links | Active |
| Mobile SDK | Active |
| Web App | Active |
| Webhooks | Active |
| Studio - Integrations | Active |
| Exchange Integrations | Active |
| Active | |
| Weather & Location Services | Active |
View the latest incidents for Movable Ink and check for official updates:
Description: The issue starting recommendations has been resolved. Please reach out to the Technical Support team if you continue to see this issue or have additional questions. A post-mortem will be posted within 7 business days.
Status: Resolved
Impact: Minor | Started At: Oct. 7, 2024, 2:47 p.m.
Description: ### Incident Summary On Friday September 27th, 2024 at approximately 7:40 PM Eastern, an internal framework upgrade failure caused issues using the recommendations service. ### Resolution Technical Support alerted the Engineering team to recommendations run failures, and Engineering discovered that a dependency inconsistency was created with code that was only required for one of our integrations. To resolve the issue, we manually downgraded to the stable version we had running. ### Corrective Actions We will be addressing the issue formally with a rebuild of this framework, deploying as part of our infrastructure-as-code initiative, which we aim to complete by the end of 2024.
Status: Postmortem
Impact: Major | Started At: Sept. 27, 2024, 10:21 p.m.
Description: ### Incident Summary On Friday September 20th, 2024 at around 11 AM Eastern a feature of the new audiences was enabled in the Da Vinci application. This exposed a bug where the new feature wasn't properly gated for accounts that still relied on the legacy behavior. A fix was deployed a few hours later, but prior to the fix being pushed, the bug caused Da Vinci to create some data extensions in Salesforce Marketing Cloud with schemas that did not have custom fields defined, which caused proofing issues throughout the day on Friday. ### Resolution As soon as the problematic code path was identified, a fix was deployed and proofing started working again for clients using legacy audiences. The fix was deployed before any sends were affected, however, in order to ensure that there would be no impact to upcoming campaigns we resynchronized any campaigns that had been modified during the period when the bug was active. Due to the timing of the fix, the Engineering team resynchronized the data on Friday night for all of the scheduled Saturday campaigns. The Technical Support team then completed resynchronization for all other potentially affected campaigns. ### Corrective Actions To prevent future occurrences, improvements will be made to alerts raised when proofing so that issues that show up in proofing can be detected and resolved more quickly. On-call playbooks and training have also been updated so that on-call staff will be able to more easily debug similar issues should they come up
Status: Postmortem
Impact: None | Started At: Sept. 19, 2024, 8:30 p.m.
Description: ### Incident Summary Between the hours of 10:30 AM and 1:15 PM EST on September 9th, 2024, users encountered an inability to run proofs through our recommendations system. The event was triggered by a code change on the morning of the 9th, which contained an update to the processing of audiences. This was prematurely deployed without a check to see if a client is using our legacy audiences system. The event was detected by our monitoring system and the Engineering team resolved the issue by reverting the code change. ### Resolution As soon as the error was recognized as being caused by a change to our code, the change was reverted, and the proof runs previously resulting in an error were restarted. We then monitored the situation for an additional 12 hours to make sure that no runs were missed, and that the issue did not continue or recur. ### Corrective Actions The Da Vinci team will be launching a program to improve the recommendations framework this month, accounting for more variability in client system usage.
Status: Postmortem
Impact: Minor | Started At: Sept. 9, 2024, 3:27 p.m.
Description: ### Incident Summary On August 12th, 2024 at approximately 12:57 PM EST, the Da Vinci Engineering team were alerted to an issue with recommendations runs being unable to start successfully. Our on-call engineers determined that this incident was caused by a lack of adequate resources available in Google Cloud Platform to fulfill the requests. The issue was resolved at 8:36 PM EST, all runs restarted and successfully completed, and no campaign sends were affected. ### Resolution As the issue was caused by a lack of resources available in Google Cloud Platform, the initial suggested resolution was to try again at a later time, when resources were differently allocated, or attempt to run the job in another zone. The Engineering team first tried to manually throttle the resource requests so that there weren't as many simultaneous attempts, which allowed us to get some of the recommendations runs started successfully. However, it became clear that this wouldn't be sufficient to resolve the issue for other affected customers, and Da Vinci Engineering made some application changes to redeploy the service in additional zones. This resolved the issue. ### Corrective actions To prevent future similar issues, Engineering made permanent application changes to redeploy the service in additional zones and allow for a better distribution of resources.
Status: Postmortem
Impact: Major | Started At: Aug. 12, 2024, 6:45 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
