-
Time: March 4, 2024, 2 a.m.
Status: Resolved
Update: This incident has been resolved. All systems are operating at optimal speeds. We appreciate your patience!
-
Time: March 3, 2024, 11:17 p.m.
Status: Monitoring
Update: WEEKEND UPDATE:
A fix has been deployed from our test environment into production. During this time, all scheduled jobs are being recycled and rescheduled and you may experience delays if you are using the application.
CURRENT STATUS:
1. All Order jobs have been cycling to Pending in less than 2 minutes
2. All Shopping Cart order and inventory syncs are flowing in the 15-minute window
3. Smart Filter triggers are firing as expected
4. Item tables are loading inventory in the default 15 minute sync window
5. Fulfillment plans and parcel labels are being produced in a timely manner depending on the size of the plan
We will continue monitoring the system throughout the evening and anticipate returning to an optimal state by early Monday morning. We appreciate your patience during this time!
-
Time: March 3, 2024, 2:22 a.m.
Status: Identified
Update: WHAT YOU CAN EXPECT TOMORROW 3/2/24:
Starting at 11am CST, the Infoplus Development team will begin deployment of the database updates from the QA environment. We are expecting the application to be unavailable from 11am CST-1pm CST. After 1pm, the application will be live, but the deployment's second half will start. This entails all processing jobs being retriggered which may cause some delays in your productivity.
EXPECTATIONS FROM THIS DEPLOYMENT:
1. All Order jobs will go from ~30 minutes on Pending back to less than 2 minutes
2. All Shopping Cart order and inventory syncs will flow in the 15 minute window
3. Smart Filter triggers will fire as expected
4. Item tables will load inventory in the default 15 mninute sync window
5. Fulfillment plans will go from ~2 hours to the normal speed depending on size of the plans
-
Time: March 2, 2024, 10:39 p.m.
Status: Identified
Update: WEEKEND UPDATE:
We will continue our efforts tomorrow with the deploy to production and send out a timeline with start and anticipated end times.
UPDATES FROM TODAY:
Our team has implemented necessary application changes today to our QA environment to ensure seamless integration with the updated database systems. This involved technical adjustments to ensure that our applications communicate effectively with the new database configurations. We’ve undertaken a thorough review of our system’s architecture to ensure it meets our high standards for reliability and scalability. While the architecture has been deemed robust, we continuously seek ways to optimize performance. A comprehensive suite of tests was conducted, covering all critical aspects of our service including order processing, fulfillment, inventory management, billing, and more. These tests are crucial for ensuring that our service remains reliable and efficient.
WHAT YOU CAN EXPECT:
We are committed to continuously improving our services and infrastructure to meet and exceed customer expectations. A deployment is scheduled to bring these improvements live, and we will provide further updates with specific timelines as we proceed. This update is part of our ongoing efforts to ensure our systems are performing at their best, offering you a reliable and efficient service.
-
Time: March 2, 2024, 12:56 a.m.
Status: Identified
Update: Our team has been working to diligently address the issues resulting from the upgrade last weekend. While we have seen improvements, we are still experiencing issues associated with the mandatory database update. Resolving this issue swiftly and effectively is our top priority.
We have discussed and decided on the best path toward restoring normal operational capacity and are building a plan to execute on that. More updates will be shared here as they materialize, such as the details, expected impact, and sequence of events.
Thanks so much for sticking with us. The patience and encouragement you have shown us is tremendously appreciated.
-
Time: Feb. 29, 2024, 8:55 p.m.
Status: Identified
Update: On Sunday, to further improve our system speed and scalability, we released a database upgrade specifically centered around increasing our job processing functionality.
Normal protocols and procedures were followed during this upgrade and several database schema settings were adjusted by default as a result of rolling out the new improvements.
Some of those schema adjustments have inversely impacted our job functionality. You may be experiencing this issue in your environment. Most commonly, our alert system indicates stuck processes or orders slowly coming off pending status. While these areas of the application are slower to respond than normal, processes are still flowing. Your data is safe, secure, and accurate - data integrity has not been affected.
We take this issue very seriously and have an “All Hands on Deck” approach to implement a solution as quickly as possible. We are committed to improving our internal and external processes to avoid this scenario in the future.
We have already made improvements internally that will provide additional advanced notifications to our customers alerting them of upcoming off-calendar releases. We are working to determine the best release window for future off-calendar releases to better serve the needs of our international customer base with minimal interruption.
We thank you for trusting us and allowing us to make this right. More information will be provided throughout this situation.
In the meantime, we are leveraging our alert notifications and manually kicking off all stalled processes. While we have several team members dedicated to this effort, if you need to report an issue with any process please REACH OUT TO SUPPORT with “Process Help Needed” in the subject line.
