TrueNas jira is incredibly slow

[flashdrive]

Hello ix,

if helpful: yes, I do confirm the slow JIRA issue.

Also I did not find another way on how to make aware that for whatever reason since a couple of months my latest posts in the are not being found using the search function with my username in it. I want to be able to find my posts again, please.

 

[Ericloewe]

I'm getting reports of the bug tracker being unavailable from @FreeNASftw, I'll let them explain in detail.

 

[FreeNASftw]

There's a lot of talk about the batch.js file in this thread but my issue was a lot simpler, most of the time a simple ping to jira.ixsystems.com would have 50-80% packet loss. No other internet traffic seemed to be affected like this.

 

[jgreco]

Yeah, they seem to be hosted at some dodgy Cogent reseller.

 

[FreeNASftw]

As requested - here is the traceroute and ping


Going down the rabbit hole of reporting TN Scale issues and I have found the Jira portal essentially unresponsive for at least a last week. I can't even get into it to report this issue. I assume this is not a common issue, otherwise it would have been resolved by now.

I have attached a traceroute output and ping showing the extensive packet loss.


1?: [LOCALHOST] pmtu 1480
1: _gateway 0.655ms
1: _gateway 0.543ms
2: 100.102.0.13 7.535ms
3: 100.102.23.254 7.936ms
4: pe.7004862-2.as58511.net 7.996ms
5: vl2.cor1.ae1.au.as58511.net 223.652ms asymm 13
6: cor1.doo.au.as58511.net 28.096ms asymm 11
7: te1-1-4.cor1.220qa.nz.as58511.net 178.407ms asymm 11
8: te2-0-26.cor1.la1.us.as58511.net 174.893ms asymm 10
9: et-0-0-0.bdr101.csla1.us.as58511.net 174.291ms
10: ae0-1701.cr7-lax2.ip4.gtt.net 176.427ms asymm 11
11: no reply
12: be3360.ccr42.lax01.atlas.cogentco.com 183.217ms asymm 19
13: be3271.ccr41.lax01.atlas.cogentco.com 179.144ms asymm 18
14: be2929.ccr21.elp01.atlas.cogentco.com 200.145ms asymm 22
15: be3047.ccr22.den01.atlas.cogentco.com 211.056ms asymm 18
16: te0-0-2-3.agr14.den01.atlas.cogentco.com 205.899ms asymm 19
17: be2166.agr11.den01.atlas.cogentco.com 203.778ms asymm 19
18: te0-0-2-3.nr12.b006545-1.den01.atlas.cogentco.com 203.961ms asymm 20
19: 38.142.161.74 213.782ms asymm 22
20: 38.142.161.74 210.410ms asymm 22
21: ??? 209.177ms reached
Resume: pmtu 1480 hops 21 back 23




ping jira.ixsystems.com
PING jira.ixsystems.com (38.109.202.235) 56(84) bytes of data.
64 bytes from 38.109.202.235: icmp_seq=1 ttl=42 time=210 ms
64 bytes from 38.109.202.235: icmp_seq=7 ttl=42 time=209 ms
64 bytes from 38.109.202.235: icmp_seq=8 ttl=42 time=209 ms
^C64 bytes from 38.109.202.235: icmp_seq=24 ttl=42 time=211 ms

--- jira.ixsystems.com ping statistics ---
24 packets transmitted, 4 received, 83.3333% packet loss, time 42139ms
rtt min/avg/max/mdev = 209.226/209.847/210.681/0.627 ms


Update Oct 5

It seems that packet loss is now 0% but there is still some strange issue going on - the reported RTT to jira.ixsystems.com is 203ms which is lengthy considering to google.com is 17ms - however each reply from jira.ixsystems.com is actually taking around 6 full seconds i.e


7 packets transmitted, 7 received, 0% packet loss, time 43823ms
rtt min/avg/max/mdev = 202.979/203.341/203.831/0.255 ms

This may be by design somehow though - the portal performance issues seem to have been resolved now.

I did manage to raise a ticket a ticket while they were having this issue and it was reported that "IT is taking over this ticket. There is a internet issue outside of IX"

Thank you

 

[jgreco]

Yeah, because your Google is probably geodirected to some nearby data center, whereas you appear to be halfway around the globe from the iX Jira server which looks like it may be in Denver(ish).

 

[FreeNASftw]

Yeah, because your Google is probably geodirected to some nearby data center, whereas you appear to be halfway around the globe from the iX Jira server which looks like it may be in Denver(ish).

Yeah... that doesn't explain the actual ping result though.

 

[jgreco]

Yeah... that doesn't explain the actual ping result though.

What's your objection?

rtt min/avg/max/mdev = 209.226/209.847/210.681/0.627 ms

is quite reasonable for transit from AU to US, allowing for modest factors on each end. The packet loss wasn't, but you said that was fixed.

 

[FreeNASftw]

What's your objection?

rtt min/avg/max/mdev = 209.226/209.847/210.681/0.627 ms

is quite reasonable for transit from AU to US, allowing for modest factors on each end. The packet loss wasn't, but you said that was fixed.

7 packets transmitted, 7 received, 0% packet loss, time 43823ms

 

[jgreco]

I have no idea what the "time" field is. That appears to be some sort of nonstandard crap. Linux perhaps? They fill their userland with bug-riddled reinvented crap.

It's saying that it sent 7 packets and it received 7 packets. If the RTT min is 209ms and the RTT max is 211ms, then it looks like return traffic is happening expeditiously.

My best guess would be that this is saying that the command ran for 44 seconds, but that could potentially include things like name resolution time. Did the command actually run for 44 seconds?

 

[FreeNASftw]

I have no idea what the "time" field is. That appears to be some sort of nonstandard crap. Linux perhaps? They fill their userland with bug-riddled reinvented crap.

It's saying that it sent 7 packets and it received 7 packets. If the RTT min is 209ms and the RTT max is 211ms, then it looks like return traffic is happening expeditiously.

My best guess would be that this is saying that the command ran for 44 seconds, but that could potentially include things like name resolution time. Did the command actually run for 44 seconds?

Yes, the "time" field is the run time of the command, and it is actually taking over 40s to return 7 results.

Certainly weird, ping from my Windows client returns normal results but on Ubuntu desktop:
If I do ping jira.ixsystems.com - I get the slow results above, if I ping the exact IP that the name resolves to (as per what the ping command itself is telling me), the results are normal, all the metrics are the same except for the time to get each response is much less.

I've only tested a half dozen other names to ping but nothing else is behaving this way.

DNS lookup doesn't take that long - clearing caches and running again makes no difference to how long this is taking.

Can't think of what could be causing this very specific issue that I've never seen before and only seems to be affecting a site/name which has had known recent problems. Perhaps it is simply some odd behaviour of the ping implementation.