7 Oct NWS Data Outage [resolved]
1189 ViewsNWS data flow was restored around 9:31 AM. Sadly, very little of the missed data was retransmitted and generally is not repairable due to lack of known archives.
NWS data flow was restored around 9:31 AM. Sadly, very little of the missed data was retransmitted and generally is not repairable due to lack of known archives.
Updated 8 Oct 9:50 AM: The maintenance window got moved to 7-10 PM CDT this evening, 8 October. I will update this news item once more information is known about timing and/or when this is complete.
Updated 8 Oct 10:30 PM: The worst of the outage from the planned network maintenance lasted between 8:55 PM and about 9:30 PM. All known issues are currently resolved, but please let me know if something is amiss!
Update 2:30 PM: Outage continues and no details yet on what is happening.
Update 2:50 PM: Data is flowing again, but still no details on what happened. I am very worried that a bunch of data will never get disseminated. Will update this news item once more is known!
Update 5:00 PM: The data flow has stabilized, but there is a mountain of missed data during the outage. Unfortunately, there is no means to repair the hole as the data appeared to be going nowhere.
Update 7:30 AM: The building housing about half of the IEM compute capacity had no power from about 12:10 AM till 4:30 AM this morning. No word yet on what happened, but this certainly was not good for IEM website health! At the moment, the only data loss was the overnight collection of some webcam imagery.
Updated 8 AM, 13 May 2026: Thankfully, DNS has been stable overnight and hopefully remains that way today. Will continue to monitor!
Updated 8:40 AM, 13 May 2026: I jinxed it and Iowa State DNS is again struggling.
Updated 10 AM, 13 May 2026: Troubles continue, no ETA on resolution.
Updated 12 PM, 13 May 2026: The pain continues. The primary impact is for inter-service communication to fail or hitting remote web services to fail. Continue to attempt hacks/workarounds to keep the plates spinning.
Updated 1 PM, 13 May 2026: ISU admins are hopeful things have stabilized.
Update 8:30 PM: A network change has been made and am monitoring to see if the recent errors have gone away. Please let me know if you are still seeing problems.
Update 9:45 PM: The problems continue, will await networking help tomorrow.
Update 16 Jan 8 AM: ISU networking folks continue to troubleshoot this problem. No ETR at the moment, so thank you for your continued patience.
Update 16 Jan 12 PM: ISU networking folks put a fix in place and initial testing looks very promising. Declaring victory for now.
Updated 8:40 AM: The load balancer admins reverted a routing change and things seem to be back to normal at the moment. Still am attempting to understand why the routing change caused adverse problems. Will update again once more is known!
Updated 12:35 PM: A brief outage happened now as an updated configuration was put in place. Monitoring for any issues!
Updated 1 Oct 9:30 AM: While there were no enduser impacts, the webfarm itself was in a degraded monitoring state. It all should be rectified now
ISU Facilities folks have a power maintenance window scheduled between 7 and 9 AM on 23 July 2024 for Agronomy Hall, but they are not sure which exact rooms in the building will not have power during this time. So the IEM website could go into a bit of degraded state when/if this happens, but most services should remain healthy. Will update this news item during the maintenance window with any updates.
Thanks for your patience.
Updated 7 AM: Still do not have clarity if the server room is about to loose power or not, but am watching closely!
Updated 9 AM: Still have power, but continue to be unsure what is happening. Will update this news item once I learn more.
Updated sometime later: This remains a mystery, but no harm no foul I guess.
Updated 9 AM: Data is flowing again, but observations valid at 7 AM are still missing. Have not seen any official word yet on what may be happening.
Updated 11:30 AM: Outage was resolved, but the 7 AM observations were lost.
Updated 6:30 AM: An update from the NWS denotes that troubleshooting continues, but there is no ETR. The primary impact to the IEM is lack of observational data from HADS, WMO_BUFR, and MADIS. Some additional grid data flows are down as well.
Updated 10:30 AM: Not much for updates. Continue to impatiently wait!
Updated 9:30 PM: Some services have been restored, but others remain unavailable. restoration work continues.
Updated 26 June 7:30 AM: Latest update as the outage continues.
Updated 27 June 7 AM: I have not seen any official statement from the NWS that this is resolved, but I am not currently aware of anything still broken from this outage, so will mark as resolved.
Update 6 PM: Data flow has returned to about normal with no known data loss at the moment. Continue to monitor.
Update 6:15 PM: The NWS finally issued some update. Things look promising to be fully resolved soon.
Update 6:50 PM: There is an unrelated outage with the boulder datacenter, which has knocked out NOMADS. I should have that worked around now.
Update 7 AM: I continue to attempt fixes of various archive holes, but some appear to be unrepairable. This is probably the last update on the outage.
Updated 9 AM: This outage continues and there continues to be not much I can do about it.
Updated 5:30 AM 17 April: The NWS has issued an update and many things remain busted. No ETRs...
Updated 8:30 AM 18 April: The latest NWS update, outage continues, no ETR.
Updated 8 AM 23 April: The outage of some NWS services continues. The latest update from the NWS is here.
Updated 10 PM 23 April: NWS denotes that this outage is now resolved.
Updated 9 PM: This outage was resolved at about 1:30 PM this afternoon.
Updated 1:40 PM Data is flowing again, monitoring....
Updated 2:40 PM Things appear to be resolved, but products like MRMS will have holes that do not get backfilled by the NWS.
Updated 9 PM: NWS restored the dataflow now and disseminated the obs backlog since 11 AM. We should be good to go now.
Update 6 PM: We are still on the struggle bus with many unhappy virtual machines within the cluster. Continue to monitor and hope for a restoration soon!
Update 8 PM: No progress to report, latency to the backend storage remains horrible. Folks continue to work on things!
Update 9:30 PM: The backend storage is back to normal operation and I am now attempting to get everything back going and fill in data holes.
Update 6 AM: Hopefully have completely recovered from this, but please let me know of any variances found.
Updated 2 Nov 2023 9 AM: Brain surgery was completed on this server to replace the suspected to be failing CPU. Will be monitoring it closely!
Updated 9:15 AM: The networking folks updated a route to move around a problematic fiber connection and things are operating normally again. Will continue to closely monitor.
Updated 30 July 10 AM: Sadly, the same issue has returned. Networking folks are looking into it.
Updated 31 July 6 AM: ISU networking folks replaced a failed optical component and data flows have returned to normal again.
Sorry about the troubles and thanks for the patience!
Update 10 AM 29 March: The NWS data flow resumed at about 9:15 AM and thankfully no data loss appears to have happened with products queued for transmission.
Update 9 PM 24 March: Oye, what a horrible mess. Firstly, thanks for your patience today and for the help the folks within Agronomy IT provided me. Most things should be running OK now, but there are about four datasets and services that are degraded. Those include MRMS data, mtarchive stuff, an auxillary mtarchive data service, and some local research data service. Two network switches were shorted out and I am down a bunch of compute capacity at the moment. In general, please do reach out to me if you find something that is not working. I have deployed a boatload of band-aids over the past 11 hours, so we shall see how many of these hold up.
Update 8 AM 27 March: Recovery efforts will continue today and there will likely be some brief outages of this website and related services as repairs continue and temporary workarounds are undone. Will itemize progress here today. Thanks again for your patience.
Update 12 PM 27 March: The MRMS service has been recovered. There was some unavoidable data loss during the 24th, but NOAA now has a AWS Archive that is much more usable than mine anyway!
Update 10 PM 27 March: The MTArchive service has almost been fully recovered and should have a small hole repaired by mid-day tomorrow. It also had some unavoidable data loss during the outage on the 24th.
Update 11 PM 27 March: The SMOS dataflow has been fixed.
Updated: Data flow was restored just after 8:15 AM. It appears missed data was retransmitted, so no data loss.
Updated 8:40 PM: The outage lasted between about 3 PM and 5:15 PM CST. Depressingly, it appears all data was lost for that period. Will see what can be done though.
Final Update: Sadly, the data from this outage is generally unrecoverable. Sigh.
Updated 6 Jan 2 PM: The NOMADS NCEP site is working and I have moved some processing over to it, so more data is flowing now than before. The NWS has updated that the College Park outage is now hoped to be fixed by Saturday morning.
Updated 10 Jan 9 AM: Everything should be back to normal now.
ISU Network Engineers plan to do network updates between 5-7 AM with possible network outages during this time. The primary impact for the IEM would be you can't reach the website and real-time data ingest may be delayed or fail.
The central storage mtarchive uses will be down for most of the day for yearly updates and patching. I have some workarounds in place to prevent data loss, but various services that use that archive will be failing. This is not too impactful for the IEM nor its services.
I'll update this news item on Tuesday as these windows are cleared and services get restored. Thanks for your patience.
Updated 9:30 AM: The network outage lasted between 5:30 and 6:10 AM this morning with no known impacts other than some lost data from live platforms, like web cams. The mtarchive downtime is now ongoing and will update this news item later today. Thanks for your patience.
Updated 11 AM: The mtarchive outage is over, but a filesystem scrub operation is ongoing and typically takes a few days to complete. The operation causes some performance issues in the iterim.
Updated 10 AM: Not much for details has been released for this outage. There are various data holes that are likely not repairable as there is no upstream source. Oh well.
A contact I have at MADIS said that the issue is upstream of them and they continue to actively request a resolution. I'll update this news item once I hear more information about this data feed.
Update 10 Dec 2020: The MADIS data flow appears to be more stable now, but the folks at NCEP IDP are unsure if the upstream issue has been fully resolved or not. They continue to monitor.
Update 14 Dec 2020: The MADIS data flow appears to have stopped again, no updates as to what is happening.
Update 29 Jan 2021: The NCEP IDP folks believe this issue to be resolved.
11:21 PM: Power is back and services are getting back on their feet. Lots of issues yet and data holes to fill.
Updated 11 Aug 12:04 AM: The IEM should be functioning again for realtime requests. The AFOS (text product) database hole should now be filled. This is the first step in repairing other datasets on this genre. The extent of the outage was from about 11 AM this morning until 10 PM this evening.
Updated 11 Aug 12:22 AM: Some ISU ITS managed virtual machines remain offline and there is cooling capacity issues in the central data center. This marginally impacts the IEM, but some archiving is not working at the moment due to it.
Updated 5:30 AM: No power at my house is creating for a slow go with getting more of the data holes repaired. Will do what I can and keep this news item updating.
Updated 10:50 AM: Maybe I now have the hole in the VTEC (NWS Watch Warning Advisory) database and Local Storm Report (LSR) database filled. It is difficult to tell with my poor network working conditions at the moment.
Updated 8:45 PM: The logistical nightmare continues and repairs to the holes is slow going. Perhaps I have the RIDGE imagery backfilled now and will soon have the NEXRAD composites repaired. It is hopeless for me at the moment to catch up on emails and twitter mentions. Thanks for your patience.
Updated 12 August 4:30 AM: I believe the hole with the ASOS/METAR data should now be filled. I am getting questions about peak gusts in Iowa for the event and I can not really answer them until I get these holes repaired. Filling the RWIS one may not be possible as I am unsure if the DOT, who is located in Ames, was able to collect data during the Ames power outage. I will reach out to them soon.
Updated 8:05 AM: Repaired NEXRAD storm attribute archive hole.
Updated 10:35 PM: Not much for updates to share. I continue to struggle with logistics due to no power and Internet at home. Thankfully, power and cooling at work have been stable.
Update 13 Aug 12:00 PM: I have power and internet at home now, so can get after this task some more!
Update 13 Aug 4:00 PM: A previous backfill of VTEC / Storm Based Warning products did not go as planned. This was cleaned up and should be all square now.
Update 16 August 2:20 PM: I have now repaired the hole with the SPC Day 1 outlooks and MCDs.
Update 10 AM: The missed RIDGE imagery has been generally fixed with the most commonly used products processed.
Thanks for your patience!
Update 8:30 AM 2 Dec 2019: Thanks to some help from a collegue onsite last night, we were able to cold restart a file server and get it back online in a degraded state. This morning, I replaced the failed hard drive and the system is back rebuilding redundancy. Everything should be functioning normally, but if you see trouble, please let me know!
Update 11 AM 13 Oct 2019: Had some trouble this morning with the secondary database being overwhelmed with connections. I am still reorganzing the backend datases to workaround some troubles found with the recent upgrade. The fun never stops with this and I am thankful for your patience.
Update 9 PM 15 Oct 2019: Things have generally been stable, but am still not running the configuration I would like. Am awaiting for some new upstream packages to be made available and will see if the performance issues go away with them.
Update 26 Sep, 7:11 AM: We are getting back to normal and will be attempting to repair various data holes while the network was out. Please let me know of any issues you are seeing.
Update 8:30 PM: After great gnashing of teeth, everything should be back to normal now. A data hole still exists that I will repair over the coming days. Thanks for your patience.
Update 22 July, 8:00 AM: Issues continue to be found and fixed, but please let me know if there is something still broken. At this point, figuring out what is broken is the toughest portion of the battle.
Update 6:21 AM: The outage was from about 5:57 AM till 6:08 AM. Sadly, the underlying issue was not resolved with this outage.
Update 9:30 AM: The servers are all back on their feet, but I have a small data hole to plug with the HADS and METAR data. Will update this news item again once that is repaired.
Update 3 PM: The SHEF and METAR holes should be repaired and no known issues remaining.
Update 9 PM: Data flow was fixed about 3 PM this afternoon.
Firstly, there will be an outage of some duration starting around 8 AM 8 Jan 2018 as a physical move happens and then network reconfiguration happens. Hopefully this outage can be limited in time, I will update this news item once done.
Secondly, the service will be available from a different IPv4 and IPv6 address. The domain name will not change, but perhaps some folks have IP based firewalls in place. The new IP addresses will be 129.186.90.3 and 2610:130:108:480::3.
This service will also default to HTTPS going forward.
Hopefully all of these changes are transparent to users. Famous last words! :)
Updated 8 Jan, 10:45 AM: Some network complications were found prior to the move starting, so the old service remains running for now until attempt number two is hopefully made later today. Thanks for your patience.
Updated 8 Jan, 2:45 PM: I believe we are up and stable with the new setup. If you are having trouble accessing, please let me know!
Update 2:20 PM: The NWS reports the issue to be resolved, but there was a significant hole that will not appear to be possible to backfill.
Resolved as of 12:07 PM.
Update 3:30 PM: Data is flowing again and the extent of the outage was from approximately 12 to 3 PM. I suspect some data is lost forever.
Here's a plot of my monitoring showing the downturn in available sites:

Updated 9 PM: Most sites are back now, but am unsure of when the full restoration will happen. There was a problem with a telco location in Omaha that caused this outage.
Update 10 PM 26 September: So firstly, an appology for the prolonged issues that occured. It did not help the situation that I was traveling and could not connect myself to the IEM servers during the outage. Full network service was restored around about noon today. On Sunday, the 1 hour initial outage in the morning was known and expected. What happened after about 11 AM was not. There was some router config issue that was preventing returning traffic from the IEM to reach the clients during this time. When these types of issues happen, I post updates to my @akrherz Twitter account.
There was a rather large METAR data outage last night due to issues at the Federal Aviation Administration (FAA). The FAA collects the airport weather station data (in METAR format) and disseminates it to the National Weather Service, from whom the IEM collects the data from. It is doubtful that I'll be able to repair the hole in the archive from this outage.
The outage lasted from just before midnight this morning to about 5:30 AM. Here's a plot from my monitoring showing global METAR station counts with the number of stations reporting within the past hour plotted.

The IEM website was very slow or unavailable for a period between about 3:20 and 4:30 PM on Monday, 27 April 2015. This was due to a cascade failure as a backup database server flooded a file server with IO requests and that slowed down another process that reads data from that server. Oye. The primary database server is about an order of magnitude faster than the backup server, so write loads the primary server generates sometimes slows down the backup server.
I am moving the backup database instance to a different disk system to prevent this from happening in the future. Thanks for your patience.
The IEM website was very slow or unresponsive between 1-2 PM on Thursday, 16 April 2015. I was sitting in a meeting at the time and did not immediately notice that bad things were happening. The issue was a cascading failure of my tilecache service due to a failure to generate N0R RADAR Composites during this time. The lack of current radar data was causing most of the incoming requests to bypass a caching layer and instead hit the mapserver backend, which was quickly overwhelmed with work.
I have made some changes to hopefully prevent this from happening again, thanks for your patience.
Update 8 PM: It took a while to get everything back on the network after ISU had DHCP server issues over night. We should be back at full strength now. Thanks for your patience.
ISU lost Internet access for a number of minutes a bit after 2 PM today. The local IT folks say a soon to be replaced router failed.
Internet for the entire ISU campus was out between 2:47 and 3:22 PM today (10 October). No word on what the issue was or if we are stable now.
4:15 PM update A network misconfiguration was made resulting in the campus wide outage. We should be stable now though.
Our local network/Internet was down between 10:05 and 10:12 AM this morning. Unsure what is going on other than a local building router is sick.
Our upstream source of NWS RADAR data is currently off the Internets, so we are having an outage of RADAR products. Have not heard any details on what the issue is other than network outage!
Update 2:30 PM: The upstream source returned at 2:26 PM and I have mostly repaired the missed data during the outage.
We lost Internet connectivity around 10:10 AM. It is not clear what is currently broken, but I have hacked around it by hard coding a network routing path. Unsure how stable this config is, but things are working again at the moment (10:45 AM).
A routine ISU network maintenance event this morning for the Agronomy building did not go as hoped and resulted in a prolonged outage between about 6:50 and 7:30 AM this morning.