Network Service News
Network Service News
2012 Service Announcements
Jan 31, 2012 - Maintenance Upgrade Completed
On Monday evening, January 30, 2012, we performed an upgrade to our Central Office equipment between the hours of 11pm and 1am January 31. Going forward, this upgrade will improve performance for our DSL and T1 customers and also allow us to more easily add additional capacity.
Feb 10, 2012 - CenturyLink Maintenance
CenturyLink has informed us that on Tuesday evening, February 14, they will be performing a maintenance operation on the ATM gateway that connects about half of our Colorado CenturyLink DSL customers to Brainstorm.
The maintenance will affect about half of our Colorado CenturyLink DSL Customers. Customers who live in Colorado and have a Qwest branded DSL modem may be affected. Customers who live in New Mexico will NOT be affected.
The maintenance will begin at midnight Tuesday and continue to 3am Wednesday morning.
During part or all of the maintenance period customers may not be able to access the internet.
Feb 10, 2012 - Hardware problems on web server
Thursday afternoon, Feb 9, we began experiencing crashes of one of our web servers, causing a number of sites, including http://www.gobrainstorm.net, to go down. This was traced to some server hardware that has been replaced, including a case fan. We apologize for the inconvenience.
Feb 14, 2012 - Ekiva Server Migration
We are in the process of migrating the Ekiva emails and websites to a new server. This is for the purpose of upgrading to newer hardware and bigger hard disks
Most of you should not notice any change. However some of you might see things like a few duplicate emails.
Update 2/15: The migration is now complete.
Feb 28, 2012 - Ekiva Server Maintenance
We have been informed by the company that hosts our Ekiva server that there will be a brief outage tomorrow.
HostForWeb Network Engineers will be performing a maintenance on its core routers located at Dupond Fabros facility in Elk Grove IL. During this maintenance there will be a small outage window that we expect to be less then 5 – 10 mins.
Date: Wednesday, February 29th, 2012
Time: 7:30 PM – 9:30 PM CST
5-10 mins outage
If you have any questions or concerns about this event, please contact us at http://support.hostforweb.com/
Update 3/30: Maintenance completed.
Mar 14, 2012 - Available Bandwidth in DW2 doubled
Equipment at the DW2 service location was upgraded, and additional T1 lines were provisioned. These steps doubled the available bandwidth and should alleviate speed constraints during peak hours.
Apr 14, 2012 - Data Center routing issues
Midafternoon on 4/12 and 4/13 our network data center (hosting email and websites) experienced connectivity problems for around 15 minutes each day, during which email and websites were unavailable. For more information see http://www.codero.com/status.
Jun 19, 2012 - Web Server problems
During the day we experienced some problems on our web servers which we attribute to a hardware upgrade last night. We hope to have all these problems resolved by tomorrow morning.
Update 6/19: We believe all issues have been resolved.
Jun 24, 2012 - Menefee Access Point down due to Mancos fire
Although details are sketchy right now, we are aware that our main internet access point in Mancos, on Menefee hill, is down. We know that the power feed was turned off yesterday, but we're not sure yet what damage has been done to the tower itself. We'll post updates as they come in.
Update 6/26: The access point is back operational now. We were told by the incident information office that substantial work was done last night to defend the towers on Menefee and that prospects look good for continued operation.
Update 6/29: Power has been turned off to the tower while the firefighters do another control burn on the slopes of Menefee. We should have more information this evening.
Update 6/30: Power is still off to the tower while fuel burnout continues.
Update 7/1: Power was restored around 12pm today and internet has been restored.
August 2, 2012 - 4 Points Maintenance
Maintenance is scheduled for the weekend of August 4 and 5. A few short outages are expected.
August 9, 2012 - Email Issues
Yesterday and today we've been experiencing some email issues. The mail queues have been processing more slowly than usual and cust. Customers will experience periodic failures to log in. Some emails have been delayed. We're working with our data center to resolve this issue as soon as possible.
Update 8/9 2:20pm: We applied a fix to the problem and so far it seems to have worked.
September 18, 2012 - Mesa Mtn Maintenance
A planned equipment move will occur between 4am and 6am on September 19
Update 9/19 5am: Completed
2013 Service Announcements
February 27, 2013 - Major network upgrade in the Telluride area
We will be performing an upgrade on our network in the Telluride area on Wednesday, February 27, around midnight. This is a major upgrade and will involve increasing bandwidth and switching our backbone sources.
We anticipate that here will be additional operations in various areas through the end of Friday.
Internet access may be unavailable for brief times during this period. We believe that most customers will not experience a significant outage.
Update 2/28 8:30am: About two thirds of customers have been converted so far.
Update 3/1 2pm: Completed.
April 18, 2013 - High winds damage on Raspberry tower repaired
On Tuesday night high winds damaged the link between the Raspberry tower and the Norwood tower. Repairs were delayed because crews first went to the Norwood tower but discovered no damage there. At the Raspberry tower, near Montrose, our crew found that the road to the National Forest was still closed so it took more time to get to the site than anticipated. Once they arrived and climbed the tower, they realized that the heavy winds had damaged the mount to the antenna. The dish was realigned, the link verified, and the mount was repaired.
May 24, 2013 - Montrose tower maintenance
On Saturday May 25, we'll be performing a brief maintenance on our equipment on the Montrose wireless tower. (Complete)
2014 Service Announcements
May 25, 2014 - Placerville outage
There is a power outage in the area which has caused our wireless Access Point to go down. We are working with San Miguel Power to restore service as quickly as possible.
June 11, 2014 - DSL Network Upgrade
We completed an upgrade to our DSL network in Colorado on Thursday and Friday June 11 and 12. This affected customers on our CenturyLink DSL service.
2015 Service Announcements
January 14, 2015 - Email Issues
Customers may experience some difficulty today connecting to email. This is due to a timing issue on our backup system. Please be aware that no emails are being lost or delayed. If you experience difficulty connecting, you may have better luck later this afternoon. We are working to resolve this problem as quickly as we can. Update Jan 15: The timing issue has been corrected, and performance is back up to par. We appreciate your patience during this period.
January 30, 2015 - Email maintenance problems
A scheduled maintenance took longer than expected resulting in some issues including delayed email and certificate problems between approximately 4pm and 7:30pm.
March 4, 2015 - Network congestion
We are currently investigating a congestion issue in our Denver Network Operations Center. You may experience intermittent difficulties with internet and voice calls. We'll update this with more information as it becomes available. UPDATE - We've isolated the problem and resolved it. The difficulties lasted about 1 1/2 hours from noon to 1:30pm.
March 5, 2015 - More network issues
Some routing problems in our Denver MPLS switch are causing more network congestion problems today. We're working with the hardware vendor to isolate and fix the problem. Meanwhile please bear with us.
We have isolated the device causing the problem.
We have removed enough of the unit to prevent it from allowing it to effect the whole MPLS ring.
We are on the phone with our vendor and they are investigating the root cause.
It is the same device that got hammered yesterday because of the routing loop.
EARLY... PRELIMINARY evidence points towards the MPLS switch getting overloaded but the jury is still out on this as a final diagnosis.
All services, as a whole, are back. There may still be individual customers having issues and we'll have to deal with them on a 1:1 basis
We have positively isolated the issue with the MPLS switch. It has a software bug and we have received the GA release that fixes this. We will have to upgrade the switch. This.... will cause us to re-boot the switch twice this afternoon. There are some services that are not running that we MUST get up. This upgrade on the MPLS switch will enable everything to stabilize and resolve any customer issues related to this outage.
All necessary updates have been applied and everything should be up, running, and stable.
March 24, 2015 - Fiber cut Farmington to Albuquerque
Our fiber to Albuquerque (provided by a third party) went down around 11:30am. There was a brief period of slow speeds and dropped connections until the routing re-converged to direct all traffic to our northern path. At this time we don't know the cause of the outage or when it will be repaired, but we understand that our provider is working to track it down. Most people won't notice this event. UPDATE: As of 1:30am March 25, the fiber cut causing the outage has been restored.
April 28, 2015 - DDOS attack against Windows web server
Our data center is currently (9:20am) being hit with a DDOS attack. About 120 web sites are affected which are hosted on our Windows server. We are currently investigating. UPDATE 11:10am - We've blocked the DDOS attack and all web sites are back up.
May 18, 2015 - Email delays and bounces
Our email database is being rebuilt this morning to correct a sync problem. This should be completed around 11am. Meanwhile, If you send an email out and get a bounce "550 Administrative prohibition" you will have to resend the email. Some incoming emails have been deferred as well and these will be delivered when the database is rebuilt.
Email is flowing again slowly. Expect delays (incoming and outgoing) for the next couple hours.
You should expect delays of an hour or less for another hour or so
All email working without delay. An upgrade to the email system is scheduled for tonight which will fix these ongoing problems.
June 15, 2015 - Email delays this morning
Our spam filtering system ran into a disk space problem at around 3:15am this morning and as a result incoming and outgoing emails were delayed. Although the problem has been resolved it will take several hours to work off the backlog, so expect delays (both incoming and outgoing) this morning. Emails from one Brainstorm address to another will be delivered instantly though.
All email is caught up now.
June 19, 2015 - Email delays this morning
Emails will be delayed this morning due to load. We expect this to last until around noon.
UPDATE 2:15pm: The disk consolidation on our VMware server is taking longer than expected. We now expect delays for another two or three hours.
UPDATE 6:00pm: The disk consolidation completed and all email is caught up.
September 23, 2015 - Smelter tower maintenance
A construction crew will be working on our Smelter Mountain tower over the next several weeks. This plan is to reinforce the tower and provide better grounding. The project requires that our equipment will need to be moved. We are working with the construction crew to reduce down time as much as possible. However, there will still be unavoidable interruptions in service.
We will post notifications with dates and times as the work is scheduled.
Friday, October 2: maintenance from noon to 4pm will cause brief interruptions in service in Durango and vicinity (Completed).
Wednesday, October 7: We've just been notified that work on our Smelter tower initiated by Verizon will result in service interruptions today in the local Durango area (outlying areas should be unaffected).
October 7, 2015 - Montoya tower power problem
A power problem at the Montoya tower on Hwy 160 is interrupting internet service in Breen and Mancos. A crew has been dispatched. ETA 4 hours (7pm tonight).
Update 10pm: Large lightning storm hit our Montoya tower late afternoon. Michael, went to Montoya’s to assess the situation and determined what equipment we needed. He had to hike in since the owner didn’t want anyone driving to the tower because it is hunting season. Michael got what he could up which included Mancos and I gathered equipment and headed to the gate to meet Michael.
Both of us headed up there, him for the second time, and it took us until about 9:00 to get it all back up again. We got back to the truck close to 10:00 by the light of our cell phones. His phone died about 3 minutes into the hike and mine died as soon as we got to the truck. : ) - Sid
November 2, 2015 - Albuquerque data path failure
7am: We've been told by our carrier to expect a two to three hour delay before the data path will be fixed. Meanwhile we're operating with half our bandwidth. You may notice slowdowns in certain applications.
1:30pm. All network transport has now been restored and we are at full capacity again.
July 22, 2016 - Norwood and Grand Junction network issues
10:40am: A failed network routing device has caused service to go down for about half our Grand Junction customers. We have an engineer en route to replace the equipment, and expect to have it repaired by early afternoon.
Update 12:20pm: Hardware has been replaced and everyone in GJ should be back up now.
We also have several customers down in Norwood due to an unrelated issue. We have three technicians making service calls to repair the issue and hope to have all customer up by the end of the day.
Yesterday, a routine network operation in the Norwood area failed and caused the "firmware" or the software that operates your internet receiver unit to become corrupted. This was due to a combination of failures which unfortunately all occurred at once.
As a result, in order to restore your service, a technician will have to come to your premises and manually reset the device. We hope to be able to get a lot of you up and running today but it looks like the operation will continue tomorrow, Saturday, June 23, as well.
Update Wednesday July 27 9:40am: About 75% of customers internet service has been restored in Norwood. It's a slow process because of the need for an on site visit. However we anticipate service will be restored to between 90% and 95% of customers by the end of today.
Update Thursday July 28 11:15am: At this time over 95% of customers who were affected by last Thursday's outage have been restored to working internet service.
Of the remaining customers who are still without service, we have scheduled repair appointments for today or tonight.
There are still a few folks we have not been able to contact for some reason and who we suspect may have been affected. Please let us know if you have not been contacted and your internet is not working. The repair process is not difficult but it does require an on site visit.
September 7, 2016 - Email Problems
2:15pm - Our email SQL backend failed since one of the disks failed. We are in the process of restoring - please bear with us. You will not lose any incoming emails but for the time being sending and receiving will not work.
2:40pm - Restoration of a backup from 9/3 is in progress. Any emails that went into quarantine since 9/3 or any changes in Roundcube settings since 9/3 will be lost.
8:30pm - Restoration complete and services restored.