This is the 3rd time in the past couple months (at most) where Comcast has done this type of maintenance in Silicon Valley. Important detail: this is cable network/node maintenance, not IP network. Every time this work has been done in the past few months, it manifests in mostly the same way. Per discussions with other list members as well as with some folks in Comcast Network Maintenance Engineering, what I see for my specific area is not necessarily what's going on a city or two over (or sometimes even within the same city!). That said, this is a "general overview" of what I've personally witnessed (with months of logs to back it up): They appear to be trying to introduce additional downstream channels. Where I live (Old Mountain View), the freqs were 735/741/747/753MHz. During the last maintenance (2-3 weeks ago) they got this working (in contrast the maintenance 3-4 weeks prior to that, which was a failure and had to be rolled back), but for Mountain View there was a period of over 24 hours of 25-30% packet loss when reaching the CMTS. Tonight's work is now a horse of a different colour: they appear to have pulled 735-753MHz and instead introduced 657MHz (?!?). So right now I've 21 bonded DS channels, which is a bit odd (pun intended), with a very elevated rate of DS correctables and occasional batches of uncorrectables. The latter is likely due to technicians still working on things (it's 02:53 PST), but the former, well... I'll just say it: There are network issues here in my area, particularly an abnormally high rate of DS correctables, i.e. RS/FEC is being applied heavily, where the pre-FEC BER rate is no where near what it was prior to July 2016. I've been in contact with Network Maintenance Engineering techs and supervisors out of Menlo Park for months about this, and met in person/on foot in the field several times. Every time they do the above work, though, the issue is exacerbated -- in almost all cases DS SNR gets substantially worse (2-3dB at times), as does DS power. It's often a specific frequency range that suffers this way. What makes this extra weird is that for a period of almost 48 hours on 9/24, the issues I describe in this paragraph *completely disappeared*, then reappeared out of no where; this leaves me with the impression there is some interconnect wiring, or hub, or *something* on the network between here and Santa Clara that got jostled in such a way where the issue went away. It really smells interference seepage. I can't speak for others, but all this is work is extra annoying for me, because every time they add new frequencies I have a boatload of scripts and DBs to alter/modify/manipulate as a result. It takes me a couple hours, and in a couple occasions right when I got everything updated, they'd make another change and I'd have to do the same again. Like for tonight, I'm actually afraid to change any of my code because for all I know in 2 hours they might change it all again. I'm fine with maintenance, but this feels much like someone "playing around in production", rather than having a plan of action and proper rollback procedure. It's more like "eh, it mostly kinda works, I guess we'll leave it like that for now", rinse lather repeat every 2-4 weeks. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | On Wed, Nov 30, 2016 at 12:28:54AM -0800, Grant Ridder via Outages wrote:
Comcast appears to be experiencing an outage in the south SF Bay Area. Not sure how wide spread it is but confirmed on the automated customer service line
Grant
Sent from my iPhone _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages