Is there an ShipHawk outage?
ShipHawk status: Systems Active
Last checked: 7 minutes ago
Get notified about any outages, downtime or incidents for ShipHawk and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 7 minutes ago
Get notified about any outages, downtime or incidents for ShipHawk and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for ShipHawk.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| ShipHawk Application | Active |
| ShipHawk Website | Active |
| Carrier/3PL Connectors | Active |
| DHL eCommerce | Active |
| FedEx Web Services | Active |
| LTL / Other Carrier Web Services | Active |
| UPS Web Services | Active |
| USPS via Endicia | Active |
| USPS via Pitney Bowes | Active |
| ShipHawk APIs | Active |
| Shipping APIs | Active |
| WMS APIs | Active |
| ShipHawk Application | Active |
| WMS | Active |
| ShipHawk Instances | Active |
| sh-default | Active |
| sh-p-2 | Active |
| System Connectors | Active |
| Acumatica App | Active |
| Amazon Web Services | Active |
| Magento | Active |
| Oracle NetSuite SuiteApp | Active |
| Shopify App | Active |
View the latest incidents for ShipHawk and check for official updates:
Description: Customer impact: ShipHawk servers are up and available, and some customers are being impacted by internet connectivity with Amazon US-East-2 resources. Here is the latest update from Amazon: "1:06 PM PST Between 11:34 AM and 12:51 PM PST, customers experienced Internet connectivity issues for some networks to and from the US-EAST-2 Region. Connectivity between instances within the Region, in between Regions, and Direct Connect connectivity were not impacted by this issue. The issue has been resolved and connectivity has been fully restored." Start Time: 11:34am Pacific Time End Time: 1:06pm Pacific Time
Status: Resolved
Impact: Minor | Started At: Dec. 5, 2022, 8 p.m.
Description: ## Incident summary Some of the ShipHawk NetSuite users experienced slowness in item fulfillments syncing between NetSuite and ShipHawk. The slowness was detected by the monitoring system at 9:28 AM Pacific Time, Monday 8/8, and continued till 12:51 PM Pacific Time. ## Impact Because of internal configuration changes, proposed shipment generation for large orders that had incomplete product information was done incorrectly and caused generation of a huge amount of packages. Processing of those proposed shipments took too much memory on background workers that were processing that queue. That, in turn, caused their unstable behavior and caused delays for all other item fulfillments processed in that queue. As a result, NetSuite Item Fulfillments were synchronizing to ShipHawk with a delay from 3 to 52 minutes. ## Detection and Recovery The incident was detected by ShipHawk monitoring system when the synchronization delay reached 3 minutes. The initial response was to scale processing power. Adding additional resources did not help as the new background job processors quickly became stuck for the same reason. The delay eventually increased and reached 52 minutes at its peak. At 12:30 PM we fixed the data of the products that were causing the issue and removed incorrectly generated proposed shipments. That unblocked the system and all the jobs that were waiting in the queue were processed within 21 minutes. The system returned to its normal state at 12:51 PM Pacific Time. ## Corrective actions In order to prevent that type of issue in the future, we plan to accomplish the following: 1. Develop a time-limiting system for background job processors, so a few slow jobs don’t block the entire queue. 2. Improve the UX to eliminate the ability to create product configurations that could cause unexpected behavior. 3. Add hard limitations to specific actions of the system, in order to reduce the risk of resource-abusive processes.
Status: Postmortem
Impact: None | Started At: Aug. 8, 2022, 5:10 p.m.
Description: ## Incident summary Some of the ShipHawk NetSuite users experienced slowness in item fulfillments syncing between NetSuite and ShipHawk. The slowness was detected by the monitoring system at 9:28 AM Pacific Time, Monday 8/8, and continued till 12:51 PM Pacific Time. ## Impact Because of internal configuration changes, proposed shipment generation for large orders that had incomplete product information was done incorrectly and caused generation of a huge amount of packages. Processing of those proposed shipments took too much memory on background workers that were processing that queue. That, in turn, caused their unstable behavior and caused delays for all other item fulfillments processed in that queue. As a result, NetSuite Item Fulfillments were synchronizing to ShipHawk with a delay from 3 to 52 minutes. ## Detection and Recovery The incident was detected by ShipHawk monitoring system when the synchronization delay reached 3 minutes. The initial response was to scale processing power. Adding additional resources did not help as the new background job processors quickly became stuck for the same reason. The delay eventually increased and reached 52 minutes at its peak. At 12:30 PM we fixed the data of the products that were causing the issue and removed incorrectly generated proposed shipments. That unblocked the system and all the jobs that were waiting in the queue were processed within 21 minutes. The system returned to its normal state at 12:51 PM Pacific Time. ## Corrective actions In order to prevent that type of issue in the future, we plan to accomplish the following: 1. Develop a time-limiting system for background job processors, so a few slow jobs don’t block the entire queue. 2. Improve the UX to eliminate the ability to create product configurations that could cause unexpected behavior. 3. Add hard limitations to specific actions of the system, in order to reduce the risk of resource-abusive processes.
Status: Postmortem
Impact: None | Started At: Aug. 8, 2022, 5:10 p.m.
Description: ## Incident summary ShipHawk API and Web Portal were not available between 9:57 AM and 11:15 AM Pacific Time, 7/28/2022. The incident was caused by an AWS outage at US-EAST-2. ## Detection This incident was detected at 10:02 AM Pacific Time when the internal alerting system diagnosed an outage. Some of the application servers, primary database node, search engine nodes were not accessible. After more investigation, we found that disk volumes attached to the primary database are completely inaccessible. Eventually, we found that it was caused by a major outage in the AWS US-EAST-2a availability zone. ## Recovery After it was confirmed that the issues are caused by the US-EAST-2a outage at 10:30 AM, the devops team initiated switching to the database replica which is located in a different AWS availability zone. That was finished at 11:09 AM and it took additional 10 minutes until all services fully recovered. ## Timeline All times are Pacific Time. 09:57 AM - the system response time started growing 10:02 AM - internal notification systems signaled about the primary database node outage 10:07 AM - the engineering team started the investigation 10:30 AM - the root cause was identified and the team started working on recovery plan 11:09 AM - the database replica was promoted to a primary node 11:19 AM - the system has fully recovered ## Corrective actions 1. Increase number of availability zones in order to minimize the effect of potential AWS outage 2. Reduce time it takes to switch to redundant availability zones.
Status: Postmortem
Impact: Critical | Started At: July 28, 2022, 5:11 p.m.
Description: This incident has been resolved. A post mortem will be made available on this status page on Friday 24 June 2022. Customer impact: Some customers were unable to log into ShipHawk during this time. Start Time: 8:10am Pacific Time End Time: 9:15am Pacific Time
Status: Resolved
Impact: Minor | Started At: June 21, 2022, 3:45 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
