Heads up: Today, as of about 2016-01-14T09:15:00Z, I'm observing high packet loss and queuing/latency (100ms) across a link between core1.pdx.he.net and core1.pao1.he.net. A report has been placed in with the HE.net NOC, but as a non-direct-customer, not seeing any traction, yet. An example MTR report from AWS in Oregon into a SF Bay Area HE path: jof@hostname:~$ mtr --report [HIGH_LATENCY_HOST] HOST: hostname Loss% Snt Last Avg Best Wrst StDev 1.|-- ec2-x-x-x-x..... 0.0% 10 0.4 0.4 0.3 0.5 0.0 2.|-- x.x.x.x 0.0% 10 0.4 0.4 0.4 0.5 0.0 3.|-- x.x.x.x 0.0% 10 0.6 0.6 0.6 0.6 0.0 4.|-- x.x.x.x 0.0% 10 1.4 1.0 0.7 1.4 0.4 5.|-- x.x.x.x 0.0% 10 7.6 7.2 6.6 7.6 0.4 6.|-- x.x.x.x 0.0% 10 10.5 12.2 7.5 17.1 3.2 7.|-- 52.95.52.11 0.0% 10 7.2 7.6 7.2 8.0 0.2 8.|-- 10ge1-12.core1.sea1.he.net 0.0% 10 6.7 21.6 6.3 37.7 10.6 9.|-- 10ge1-1.core1.pdx1.he.net 0.0% 10 10.8 11.5 10.6 15.4 1.6 10.|-- 10ge12-7.core1.pao1.he.net 30.0% 10 109.2 101.9 95.2 111.5 6.2 11.|-- 10ge4-4.core3.fmt2.he.net 20.0% 10 112.0 117.9 95.7 169.5 27.8 12.|-- x.x.x.x 0.0% 10 103.7 101.8 96.7 105.6 2.6 13.|-- x.x.x.x 20.0% 10 106.1 106.0 102.9 111.4 2.7 Cheers, jof
I'm seeing the same to some of my customers in southern Oregon and Northern California. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Jonathan Lassoff via Outages Sent: Thursday, January 14, 2016 10:54 AM To: Outages Mailing List <outages@outages.org> Subject: [outages] High HE.net PacketLoss between PDX<->PAO Heads up: Today, as of about 2016-01-14T09:15:00Z, I'm observing high packet loss and queuing/latency (100ms) across a link between core1.pdx.he.net and core1.pao1.he.net. A report has been placed in with the HE.net NOC, but as a non-direct-customer, not seeing any traction, yet. An example MTR report from AWS in Oregon into a SF Bay Area HE path: jof@hostname:~$ mtr --report [HIGH_LATENCY_HOST] HOST: hostname Loss% Snt Last Avg Best Wrst StDev 1.|-- ec2-x-x-x-x..... 0.0% 10 0.4 0.4 0.3 0.5 0.0 2.|-- x.x.x.x 0.0% 10 0.4 0.4 0.4 0.5 0.0 3.|-- x.x.x.x 0.0% 10 0.6 0.6 0.6 0.6 0.0 4.|-- x.x.x.x 0.0% 10 1.4 1.0 0.7 1.4 0.4 5.|-- x.x.x.x 0.0% 10 7.6 7.2 6.6 7.6 0.4 6.|-- x.x.x.x 0.0% 10 10.5 12.2 7.5 17.1 3.2 7.|-- 52.95.52.11 0.0% 10 7.2 7.6 7.2 8.0 0.2 8.|-- 10ge1-12.core1.sea1.he.net 0.0% 10 6.7 21.6 6.3 37.7 10.6 9.|-- 10ge1-1.core1.pdx1.he.net 0.0% 10 10.8 11.5 10.6 15.4 1.6 10.|-- 10ge12-7.core1.pao1.he.net 30.0% 10 109.2 101.9 95.2 111.5 6.2 11.|-- 10ge4-4.core3.fmt2.he.net 20.0% 10 112.0 117.9 95.7 169.5 27.8 12.|-- x.x.x.x 0.0% 10 103.7 101.8 96.7 105.6 2.6 13.|-- x.x.x.x 20.0% 10 106.1 106.0 102.9 111.4 2.7 Cheers, jof _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages Attention: Information contained in this message and or attachments is intended only for the recipient(s) named above and may contain confidential and or privileged material that is protected under State or Federal law. If you are not the intended recipient, any disclosure, copying, distribution or action taken on it is prohibited. If you believe you have received this email in error, please contact the sender with a copy to compliance@ochin.org, delete this email and destroy all copies.
Here is what I'm seeing from HE PDX->PAO # mtr --report -c 20 core1.pao1.he.net HOST: noc.cascadetech.org Loss% Snt Last Avg Best Wrst StDev 1. agr1.ptk.net.cascadetech.org 0.0% 20 2.0 1.2 0.9 2.3 0.4 2. as-ptk.ae1-2.agr1.ptk.networ 0.0% 20 0.2 1.4 0.2 11.3 3.0 3. 10ge3-3.core1.pdx1.he.net 0.0% 20 4.5 6.1 0.2 11.4 3.6 4. core1.pao1.he.net 0.0% 20 14.5 15.3 14.5 19.6 1.4 On Thu, Jan 14, 2016 at 10:53 AM, Jonathan Lassoff via Outages < outages@outages.org> wrote:
Heads up: Today, as of about 2016-01-14T09:15:00Z, I'm observing high packet loss and queuing/latency (100ms) across a link between core1.pdx.he.net and core1.pao1.he.net.
A report has been placed in with the HE.net NOC, but as a non-direct-customer, not seeing any traction, yet.
An example MTR report from AWS in Oregon into a SF Bay Area HE path:
jof@hostname:~$ mtr --report [HIGH_LATENCY_HOST] HOST: hostname Loss% Snt Last Avg Best Wrst StDev 1.|-- ec2-x-x-x-x..... 0.0% 10 0.4 0.4 0.3 0.5 0.0 2.|-- x.x.x.x 0.0% 10 0.4 0.4 0.4 0.5 0.0 3.|-- x.x.x.x 0.0% 10 0.6 0.6 0.6 0.6 0.0 4.|-- x.x.x.x 0.0% 10 1.4 1.0 0.7 1.4 0.4 5.|-- x.x.x.x 0.0% 10 7.6 7.2 6.6 7.6 0.4 6.|-- x.x.x.x 0.0% 10 10.5 12.2 7.5 17.1 3.2 7.|-- 52.95.52.11 0.0% 10 7.2 7.6 7.2 8.0 0.2 8.|-- 10ge1-12.core1.sea1.he.net 0.0% 10 6.7 21.6 6.3 37.7 10.6 9.|-- 10ge1-1.core1.pdx1.he.net 0.0% 10 10.8 11.5 10.6 15.4 1.6 10.|-- 10ge12-7.core1.pao1.he.net 30.0% 10 109.2 101.9 95.2 111.5 6.2 11.|-- 10ge4-4.core3.fmt2.he.net 20.0% 10 112.0 117.9 95.7 169.5 27.8 12.|-- x.x.x.x 0.0% 10 103.7 101.8 96.7 105.6 2.6 13.|-- x.x.x.x 20.0% 10 106.1 106.0 102.9 111.4 2.7
Cheers, jof _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Eric Harrison Network Services Cascade Technology Alliance / Multnomah Education Service District office: 503-257-1554 cell: 971-998-6249 NOC 503-257-1510
Looks like this was a backup path being saturated when the normal SEA<->SJC was down. Apparently, there was an outage in San Jose earlier, and all is well now. Thanks, jof On Thu, Jan 14, 2016 at 11:05 AM, Eric Harrison <eric.harrison@cascadetech.org> wrote:
Here is what I'm seeing from HE PDX->PAO
# mtr --report -c 20 core1.pao1.he.net HOST: noc.cascadetech.org Loss% Snt Last Avg Best Wrst StDev 1. agr1.ptk.net.cascadetech.org 0.0% 20 2.0 1.2 0.9 2.3 0.4 2. as-ptk.ae1-2.agr1.ptk.networ 0.0% 20 0.2 1.4 0.2 11.3 3.0 3. 10ge3-3.core1.pdx1.he.net 0.0% 20 4.5 6.1 0.2 11.4 3.6 4. core1.pao1.he.net 0.0% 20 14.5 15.3 14.5 19.6 1.4
On Thu, Jan 14, 2016 at 10:53 AM, Jonathan Lassoff via Outages <outages@outages.org> wrote:
Heads up: Today, as of about 2016-01-14T09:15:00Z, I'm observing high packet loss and queuing/latency (100ms) across a link between core1.pdx.he.net and core1.pao1.he.net.
A report has been placed in with the HE.net NOC, but as a non-direct-customer, not seeing any traction, yet.
An example MTR report from AWS in Oregon into a SF Bay Area HE path:
jof@hostname:~$ mtr --report [HIGH_LATENCY_HOST] HOST: hostname Loss% Snt Last Avg Best Wrst StDev 1.|-- ec2-x-x-x-x..... 0.0% 10 0.4 0.4 0.3 0.5 0.0 2.|-- x.x.x.x 0.0% 10 0.4 0.4 0.4 0.5 0.0 3.|-- x.x.x.x 0.0% 10 0.6 0.6 0.6 0.6 0.0 4.|-- x.x.x.x 0.0% 10 1.4 1.0 0.7 1.4 0.4 5.|-- x.x.x.x 0.0% 10 7.6 7.2 6.6 7.6 0.4 6.|-- x.x.x.x 0.0% 10 10.5 12.2 7.5 17.1 3.2 7.|-- 52.95.52.11 0.0% 10 7.2 7.6 7.2 8.0 0.2 8.|-- 10ge1-12.core1.sea1.he.net 0.0% 10 6.7 21.6 6.3 37.7 10.6 9.|-- 10ge1-1.core1.pdx1.he.net 0.0% 10 10.8 11.5 10.6 15.4 1.6 10.|-- 10ge12-7.core1.pao1.he.net 30.0% 10 109.2 101.9 95.2 111.5 6.2 11.|-- 10ge4-4.core3.fmt2.he.net 20.0% 10 112.0 117.9 95.7 169.5 27.8 12.|-- x.x.x.x 0.0% 10 103.7 101.8 96.7 105.6 2.6 13.|-- x.x.x.x 20.0% 10 106.1 106.0 102.9 111.4 2.7
Cheers, jof _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Eric Harrison Network Services Cascade Technology Alliance / Multnomah Education Service District office: 503-257-1554 cell: 971-998-6249 NOC 503-257-1510
participants (3)
-
Eric Harrison -
Jonathan Lassoff -
Tyler Applebaum