Ξ welcome to cryptostorm's member forums ~ you don't have to be a cryptostorm member to post here Ξ
Ξ any OpenVPN configs found on the forum are likely outdated. For the latest, visit here or GitHub Ξ
Ξ If you're looking for tutorials/guides, check out the new https://cryptostorm.is/#section6 Ξ

Serious Drops and packet loss South node

Looking for assistance with a cryptostorm connection issue? Post here & we'll help out. Also: if you're not sure where to post, do so here & we'll move things around as needed. Also: for quickest support, email our oddly calm & easygoing support reps at support@cryptostorm.is :)

Topic Author
NOYB

Serious Drops and packet loss South node

Postby NOYB » Mon Oct 31, 2016 8:20 pm

Connection once would stay up for days....now dies in minutes. This problem is ongoing - unrelated to any specific time of day.

Noticed one of your hops is labeled "Ubiquiti". It takes extraordinary engineering to yield a reliable microwave link within urban areas using unlicensed spectrum. I hope you haven't been conned.
Attachments
Cryptostorm Problems.png

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Tue Nov 01, 2016 2:19 am

@OP

Looking at that image, I'm inclined to agree - the (possible) giveaway is the large ping compared to the rest of the trace. Having said that, I visited the Ubiquity.IO website and saw nothing re: microwave links. It could be that the routing out of the DC uses Ubiquity.io for transit and that network segment is running hot.

It might be worth hopping on IRC and giving df a nudge.

EDIT
What's the connection like with other nodes? Is the south node the closest one to your location?


Topic Author
NOYB

Re: Serious Drops and packet loss South node

Postby NOYB » Tue Nov 01, 2016 5:03 am

Fixed it.

Bought AirVPN service and installed on our proxy. No more drops. Kinda suspected that would be the case, because while Frontier Fios is populated by people who mean no good - their service has been pretty reliable.

"Substitution" was the cleanest most concrete way to prove causality. I don't wish to "move" - but what choice are you giving me?

The other nodes are dropping too. While I *expect* to reset our proxy say every two weeks (crytostorm norm)...these every ten minute drops have me scurrying for the exit. Just not good enough.

We use cryptostorm to evade these damn deep packet inspection policies just about every ISP practices...but cryptostorm is proving useless now. Started about 2-3 weeks ago and has never let up.


Topic Author
r5rr

Re: Serious Drops and packet loss South node

Postby r5rr » Tue Nov 01, 2016 5:59 pm

Thanks for the updates dns resolver listing df and team.

need help with accessibility for RO-RU voodoo node


Topic Author
Krin

Re: Serious Drops and packet loss South node

Postby Krin » Thu Nov 10, 2016 1:08 pm

Same issue. It's been a month or two for me. Connection drops within minutes (usually 5-10 minutes) and won't reconnect. Using Tunnelblick 3.6.8 (build 4625) on Mac OS 10.12.1. Tried North, South, Central. BTW, does anyone actually have specific info on where those nodes are?


Khariz
Posts: 162
Joined: Sun Jan 17, 2016 7:48 am

Re: Serious Drops and packet loss South node

Postby Khariz » Fri Nov 11, 2016 10:03 am

I've been connected to south for 72 hours straight without any drops or significant spikes. Not sure what to tell you.


Topic Author
NOYB

Re: Serious Drops and packet loss South node

Postby NOYB » Sun Nov 13, 2016 2:19 am

Have 11 months remaining on this CS token, perhaps half month remaining on AirVPN.


But AirVPN is still up :(

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Tue Nov 15, 2016 5:40 am

@Krin

It sounds like there's a routing issue between you and the South node; it appears to be reinforced by the fact that Khariz appears to have no issues. Can you do a traceroute and post the results here?

Thanks. :)


Topic Author
NOYB

Re: Serious Drops and packet loss South node

Postby NOYB » Fri Nov 18, 2016 1:30 am

Tried again. Stayed up perhaps 10 minutes. Back to AirVPN .....

Code: Select all

Thu Nov 17 13:55:25 2016 us=348059 Current Parameter Settings:
Thu Nov 17 13:55:25 2016 us=348059   config = '..\user\custom.conf'
Thu Nov 17 13:55:25 2016 us=348059   mode = 0
Thu Nov 17 13:55:25 2016 us=348059 NOTE: --mute triggered...
Thu Nov 17 13:55:25 2016 us=348059 357 variation(s) on previous 3 message(s) suppressed by --mute
Thu Nov 17 13:55:25 2016 us=348059 OpenVPN 2.3.6 i686-w64-mingw32 [SSL (OpenSSL)] [LZO] [PKCS11] [IPv6] built on Dec  1 2014
Thu Nov 17 13:55:25 2016 us=348059 library versions: OpenSSL 1.0.2 22 Jan 2015, LZO 2.08
Thu Nov 17 13:55:27 2016 us=130622 LZO compression initialized
Thu Nov 17 13:55:27 2016 us=130622 Control Channel MTU parms [ L:1604 D:140 EF:40 EB:0 ET:0 EL:0 ]
Thu Nov 17 13:55:27 2016 us=150651 Socket Buffers: R=[8192->8192] S=[8192->8192]
Thu Nov 17 13:55:27 2016 us=240780 Data Channel MTU parms [ L:1604 D:1450 EF:104 EB:135 ET:0 EL:0 AF:3/1 ]
Thu Nov 17 13:55:27 2016 us=240780 Local Options String: 'V4,dev-type tun,link-mtu 1604,tun-mtu 1500,proto TCPv4_CLIENT,comp-lzo,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-client'
Thu Nov 17 13:55:27 2016 us=240780 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1604,tun-mtu 1500,proto TCPv4_SERVER,comp-lzo,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-server'
Thu Nov 17 13:55:27 2016 us=240780 Local Options hash (VER=V4): '06d8c75c'
Thu Nov 17 13:55:27 2016 us=240780 Expected Remote Options hash (VER=V4): 'b20ffe30'
Thu Nov 17 13:55:27 2016 us=240780 Attempting to establish TCP connection with [AF_INET]70.32.38.69:443 [nonblock]
Thu Nov 17 13:55:28 2016 us=242220 TCP connection established with [AF_INET]70.32.38.69:443
Thu Nov 17 13:55:28 2016 us=242220 TCPv4_CLIENT link local: [undef]
Thu Nov 17 13:55:28 2016 us=242220 TCPv4_CLIENT link remote: [AF_INET]70.32.38.69:443
Thu Nov 17 13:55:28 2016 us=242220 TLS: Initial packet from [AF_INET]70.32.38.69:443, sid=30927b06 545526f0
Thu Nov 17 13:55:28 2016 us=252235 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Thu Nov 17 13:55:28 2016 us=332350 VERIFY OK: depth=1, C=CA, ST=QC, L=Montreal, O=Katana Holdings Limite /  cryptostorm_darknet, OU=Tech Ops, CN=cryptostorm_is, emailAddress=certadmin@cryptostorm.is
Thu Nov 17 13:55:28 2016 us=332350 VERIFY OK: nsCertType=SERVER
Thu Nov 17 13:55:28 2016 us=332350 VERIFY OK: depth=0, C=CA, ST=QC, L=Montreal, O=Katana Holdings Limite /  cryptostorm_darknet, OU=Tech Ops, CN=server, emailAddress=certadmin@cryptostorm.is
Thu Nov 17 13:55:29 2016 us=534078 NOTE: --mute triggered...
Thu Nov 17 13:55:29 2016 us=534078 5 variation(s) on previous 3 message(s) suppressed by --mute
Thu Nov 17 13:55:29 2016 us=534078 [server] Peer Connection Initiated with [AF_INET]70.32.38.69:443
Thu Nov 17 13:55:31 2016 us=757275 SENT CONTROL [server]: 'PUSH_REQUEST' (status=1)
Thu Nov 17 13:55:31 2016 us=967577 PUSH: Received control message: 'PUSH_REPLY,persist-key,persist-tun,redirect-gateway def1,redirect-gateway bypass-dhcp,dhcp-option DNS 70.32.38.67,route-gateway 10.45.0.1,topology subnet,ping 20,ping-restart 60,ifconfig 10.45.206.219 255.255.0.0'
Thu Nov 17 13:55:31 2016 us=967577 OPTIONS IMPORT: timers and/or timeouts modified
Thu Nov 17 13:55:31 2016 us=967577 NOTE: --mute triggered...
Thu Nov 17 13:55:32 2016 us=67721 5 variation(s) on previous 3 message(s) suppressed by --mute
Thu Nov 17 13:55:32 2016 us=67721 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Thu Nov 17 13:55:32 2016 us=77736 open_tun, tt->ipv6=0
Thu Nov 17 13:55:32 2016 us=77736 TAP-WIN32 device [Local Area Connection 2] opened: \\.\Global\{5899D730-1570-453C-A79C-EFC4C00421E9}.tap
Thu Nov 17 13:55:32 2016 us=77736 TAP-Windows Driver Version 9.21
Thu Nov 17 13:55:32 2016 us=77736 TAP-Windows MTU=1500
Thu Nov 17 13:55:32 2016 us=87750 Set TAP-Windows TUN subnet mode network/local/netmask = 10.45.0.0/10.45.206.219/255.255.0.0 [SUCCEEDED]
Thu Nov 17 13:55:32 2016 us=87750 Notified TAP-Windows driver to set a DHCP IP/netmask of 10.45.206.219/255.255.0.0 on interface {5899D730-1570-453C-A79C-EFC4C00421E9} [DHCP-serv: 10.45.255.254, lease-time: 31536000]
Thu Nov 17 13:55:32 2016 us=87750 DHCP option string: 06044620 2643
Thu Nov 17 13:55:32 2016 us=87750 Successful ARP Flush on interface [16] {5899D730-1570-453C-A79C-EFC4C00421E9}
Thu Nov 17 13:55:37 2016 us=215123 TEST ROUTES: 1/1 succeeded len=0 ret=1 a=0 u/d=up
Thu Nov 17 13:55:37 2016 us=215123 C:\Windows\system32\route.exe ADD 70.32.38.69 MASK 255.255.255.255 192.168.1.1
Thu Nov 17 13:55:37 2016 us=245166 ROUTE: CreateIpForwardEntry succeeded with dwForwardMetric1=10 and dwForwardType=4
Thu Nov 17 13:55:37 2016 us=245166 Route addition via IPAPI succeeded [adaptive]
Thu Nov 17 13:55:37 2016 us=245166 C:\Windows\system32\route.exe ADD 192.168.1.1 MASK 255.255.255.255 192.168.1.1 IF 11
Thu Nov 17 13:55:37 2016 us=245166 ROUTE: CreateIpForwardEntry succeeded with dwForwardMetric1=10 and dwForwardType=4
Thu Nov 17 13:55:37 2016 us=245166 Route addition via IPAPI succeeded [adaptive]
Thu Nov 17 13:55:37 2016 us=245166 C:\Windows\system32\route.exe ADD 0.0.0.0 MASK 128.0.0.0 10.45.0.1
Thu Nov 17 13:55:37 2016 us=255180 ROUTE: CreateIpForwardEntry succeeded with dwForwardMetric1=20 and dwForwardType=4
Thu Nov 17 13:55:37 2016 us=255180 Route addition via IPAPI succeeded [adaptive]
Thu Nov 17 13:55:37 2016 us=255180 C:\Windows\system32\route.exe ADD 128.0.0.0 MASK 128.0.0.0 10.45.0.1
Thu Nov 17 13:55:37 2016 us=265195 ROUTE: CreateIpForwardEntry succeeded with dwForwardMetric1=20 and dwForwardType=4
Thu Nov 17 13:55:37 2016 us=265195 Route addition via IPAPI succeeded [adaptive]
Thu Nov 17 13:55:37 2016 us=265195 Initialization Sequence Completed
Connected
Thu Nov 17 14:15:29 2016 us=920148 VERIFY OK: depth=1, C=CA, ST=QC, L=Montreal, O=Katana Holdings Limite /  cryptostorm_darknet, OU=Tech Ops, CN=cryptostorm_is, emailAddress=certadmin@cryptostorm.is
Thu Nov 17 14:15:29 2016 us=920148 VERIFY OK: nsCertType=SERVER
Thu Nov 17 14:15:29 2016 us=920148 VERIFY OK: depth=0, C=CA, ST=QC, L=Montreal, O=Katana Holdings Limite /  cryptostorm_darknet, OU=Tech Ops, CN=server, emailAddress=certadmin@cryptostorm.is
Thu Nov 17 14:15:31 2016 us=222020 NOTE: --mute triggered...
Thu Nov 17 14:17:44 2016 us=123123 5 variation(s) on previous 3 message(s) suppressed by --mute
Thu Nov 17 14:17:44 2016 us=123123 Connection reset, restarting [0]
Thu Nov 17 14:17:44 2016 us=123123 TCP/UDP: Closing socket
Thu Nov 17 14:17:44 2016 us=123123 SIGUSR1[soft,connection-reset] received, process restarting
Thu Nov 17 14:17:44 2016 us=123123 Restart pause, 5 second(s)
Thu Nov 17 14:17:49 2016 us=130323 Re-using SSL/TLS context
Thu Nov 17 14:17:49 2016 us=130323 LZO compression initialized
Thu Nov 17 14:17:49 2016 us=130323 Control Channel MTU parms [ L:1604 D:140 EF:40 EB:0 ET:0 EL:0 ]
Thu Nov 17 14:17:49 2016 us=130323 Socket Buffers: R=[8192->8192] S=[8192->8192]
Thu Nov 17 14:18:01 2016 us=147603 RESOLVE: Cannot resolve host address: windows-ussouth.cstorm.pw: The requested name is valid, but no data of the requested type was found.
Thu Nov 17 14:18:01 2016 us=147603 Data Channel MTU parms [ L:1604 D:1450 EF:104 EB:135 ET:0 EL:0 AF:3/1 ]
Thu Nov 17 14:18:01 2016 us=147603 Local Options String: 'V4,dev-type tun,link-mtu 1604,tun-mtu 1500,proto TCPv4_CLIENT,comp-lzo,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-client'
Thu Nov 17 14:18:01 2016 us=147603 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1604,tun-mtu 1500,proto TCPv4_SERVER,comp-lzo,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-server'
Thu Nov 17 14:18:01 2016 us=147603 Local Options hash (VER=V4): '06d8c75c'
Thu Nov 17 14:18:01 2016 us=147603 Expected Remote Options hash (VER=V4): 'b20ffe30'
Thu Nov 17 14:18:13 2016 us=164883 RESOLVE: Cannot resolve host address: windows-ussouth.cstorm.pw: The requested name is valid, but no data of the requested type was found.
Thu Nov 17 14:18:13 2016 us=164883 TCP/UDP: Closing socket
Thu Nov 17 14:18:13 2016 us=164883 SIGUSR1[soft,init_instance] received, process restarting
Thu Nov 17 14:18:13 2016 us=164883 Restart pause, 5 second(s)
Thu Nov 17 14:18:18 2016 us=172083 Re-using SSL/TLS context
Thu Nov 17 14:18:18 2016 us=172083 LZO compression initialized
Thu Nov 17 14:18:18 2016 us=172083 Control Channel MTU parms [ L:1604 D:140 EF:40 EB:0 ET:0 EL:0 ]
Thu Nov 17 14:18:18 2016 us=172083 Socket Buffers: R=[8192->8192] S=[8192->8192]
Thu Nov 17 14:18:30 2016 us=189363 RESOLVE: Cannot resolve host address: windows-ussouth.cryptostorm.net: The requested name is valid, but no data of the requested type was found.
Thu Nov 17 14:18:30 2016 us=189363 Data Channel MTU parms [ L:1604 D:1450 EF:104 EB:135 ET:0 EL:0 AF:3/1 ]
Thu Nov 17 14:18:30 2016 us=189363 Local Options String: 'V4,dev-type tun,link-mtu 1604,tun-mtu 1500,proto TCPv4_CLIENT,comp-lzo,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-client'
Thu Nov 17 14:18:30 2016 us=189363 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1604,tun-mtu 1500,proto TCPv4_SERVER,comp-lzo,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-server'
Thu Nov 17 14:18:30 2016 us=189363 Local Options hash (VER=V4): '06d8c75c'
Thu Nov 17 14:18:30 2016 us=189363 Expected Remote Options hash (VER=V4): 'b20ffe30'
Thu Nov 17 14:18:42 2016 us=206643 RESOLVE: Cannot resolve host address: windows-ussouth.cryptostorm.net: The requested name is valid, but no data of the requested type was found.
Thu Nov 17 14:18:42 2016 us=206643 TCP/UDP: Closing socket
Thu Nov 17 14:18:42 2016 us=206643 SIGUSR1[soft,init_instance] received, process restarting
Thu Nov 17 14:18:42 2016 us=206643 Restart pause, 5 second(s)
Thu Nov 17 14:18:47 2016 us=213843 Re-using SSL/TLS context
Thu Nov 17 14:18:47 2016 us=213843 LZO compression initialized
Thu Nov 17 14:18:47 2016 us=213843 Control Channel MTU parms [ L:1604 D:140 EF:40 EB:0 ET:0 EL:0 ]
Thu Nov 17 14:18:47 2016 us=213843 Socket Buffers: R=[8192->8192] S=[8192->8192]
Thu Nov 17 14:18:59 2016 us=231123 RESOLVE: Cannot resolve host address: windows-ussouth.cryptostorm.org: The requested name is valid, but no data of the requested type was found.
Thu Nov 17 14:18:59 2016 us=231123 Data Channel MTU parms [ L:1604 D:1450 EF:104 EB:135 ET:0 EL:0 AF:3/1 ]
Thu Nov 17 14:18:59 2016 us=231123 Local Options String: 'V4,dev-type tun,link-mtu 1604,tun-mtu 1500,proto TCPv4_CLIENT,comp-lzo,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-client'
Thu Nov 17 14:18:59 2016 us=231123 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1604,tun-mtu 1500,proto TCPv4_SERVER,comp-lzo,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-server'
Thu Nov 17 14:18:59 2016 us=231123 Local Options hash (VER=V4): '06d8c75c'
Thu Nov 17 14:18:59 2016 us=231123 Expected Remote Options hash (VER=V4): 'b20ffe30'
Thu Nov 17 14:19:11 2016 us=248403 RESOLVE: Cannot resolve host address: windows-ussouth.cryptostorm.org: The requested name is valid, but no data of the requested type was found.
Thu Nov 17 14:19:11 2016 us=248403 TCP/UDP: Closing socket
Thu Nov 17 14:19:11 2016 us=248403 SIGUSR1[soft,init_instance] received, process restarting
Thu Nov 17 14:19:11 2016 us=248403 Restart pause, 5 second(s)
Thu Nov 17 14:19:16 2016 us=255603 Re-using SSL/TLS context
Thu Nov 17 14:19:16 2016 us=255603 LZO compression initialized
Thu Nov 17 14:19:16 2016 us=255603 Control Channel MTU parms [ L:1604 D:140 EF:40 EB:0 ET:0 EL:0 ]
Thu Nov 17 14:19:16 2016 us=255603 Socket Buffers: R=[8192->8192] S=[8192->8192]
Thu Nov 17 14:19:28 2016 us=272883 RESOLVE: Cannot resolve host address: windows-ussouth.cstorm.pw: The requested name is valid, but no data of the requested type was found.
Thu Nov 17 14:19:28 2016 us=272883 Data Channel MTU parms [ L:1604 D:1450 EF:104 EB:135 ET:0 EL:0 AF:3/1 ]
Thu Nov 17 14:19:28 2016 us=272883 Local Options String: 'V4,dev-type tun,link-mtu 1604,tun-mtu 1500,proto TCPv4_CLIENT,comp-lzo,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-client'
Thu Nov 17 14:19:28 2016 us=272883 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1604,tun-mtu 1500,proto TCPv4_SERVER,comp-lzo,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-server'
Thu Nov 17 14:19:28 2016 us=272883 Local Options hash (VER=V4): '06d8c75c'
Thu Nov 17 14:19:28 2016 us=272883 Expected Remote Options hash (VER=V4): 'b20ffe30'
Thu Nov 17 14:19:40 2016 us=290163 RESOLVE: Cannot resolve host address: windows-ussouth.cstorm.pw: The requested name is valid, but no data of the requested type was found.
Thu Nov 17 14:19:40 2016 us=290163 TCP/UDP: Closing socket
Thu Nov 17 14:19:40 2016 us=290163 SIGUSR1[soft,init_instance] received, process restarting
Thu Nov 17 14:19:40 2016 us=290163 Restart pause, 5 second(s)
Thu Nov 17 14:19:45 2016 us=297363 Re-using SSL/TLS context
Thu Nov 17 14:19:45 2016 us=297363 LZO compression initialized
Thu Nov 17 14:19:45 2016 us=297363 Control Channel MTU parms [ L:1604 D:140 EF:40 EB:0 ET:0 EL:0 ]
Thu Nov 17 14:19:45 2016 us=297363 Socket Buffers: R=[8192->8192] S=[8192->8192]
Thu Nov 17 14:19:57 2016 us=314643 RESOLVE: Cannot resolve host address: windows-ussouth.cryptostorm.net: The requested name is valid, but no data of the requested type was found.
Thu Nov 17 14:19:57 2016 us=314643 Data Channel MTU parms [ L:1604 D:1450 EF:104 EB:135 ET:0 EL:0 AF:3/1 ]
Thu Nov 17 14:19:57 2016 us=314643 Local Options String: 'V4,dev-type tun,link-mtu 1604,tun-mtu 1500,proto TCPv4_CLIENT,comp-lzo,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-client'
Thu Nov 17 14:19:57 2016 us=314643 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1604,tun-mtu 1500,proto TCPv4_SERVER,comp-lzo,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-server'
Thu Nov 17 14:19:57 2016 us=314643 Local Options hash (VER=V4): '06d8c75c'
Thu Nov 17 14:19:57 2016 us=314643 Expected Remote Options hash (VER=V4): 'b20ffe30'
Thu Nov 17 14:20:09 2016 us=331923 RESOLVE: Cannot resolve host address: windows-ussouth.cryptostorm.net: The requested name is valid, but no data of the requested type was found.
Thu Nov 17 14:20:09 2016 us=331923 TCP/UDP: Closing socket
Thu Nov 17 14:20:09 2016 us=331923 SIGUSR1[soft,init_instance] received, process restarting
Thu Nov 17 14:20:09 2016 us=331923 Restart pause, 5 second(s)
Thu Nov 17 14:20:14 2016 us=339123 Re-using SSL/TLS context
Thu Nov 17 14:20:14 2016 us=339123 LZO compression initialized
Thu Nov 17 14:20:14 2016 us=339123 Control Channel MTU parms [ L:1604 D:140 EF:40 EB:0 ET:0 EL:0 ]
Thu Nov 17 14:20:14 2016 us=339123 Socket Buffers: R=[8192->8192] S=[8192->8192]
Thu Nov 17 14:20:26 2016 us=356403 RESOLVE: Cannot resolve host address: windows-ussouth.cryptostorm.org: The requested name is valid, but no data of the requested type was found.
Thu Nov 17 14:20:26 2016 us=356403 Data Channel MTU parms [ L:1604 D:1450 EF:104 EB:135 ET:0 EL:0 AF:3/1 ]
Thu Nov 17 14:20:26 2016 us=356403 Local Options String: 'V4,dev-type tun,link-mtu 1604,tun-mtu 1500,proto TCPv4_CLIENT,comp-lzo,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-client'
Thu Nov 17 14:20:26 2016 us=356403 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1604,tun-mtu 1500,proto TCPv4_SERVER,comp-lzo,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-server'
Thu Nov 17 14:20:26 2016 us=356403 Local Options hash (VER=V4): '06d8c75c'
Thu Nov 17 14:20:26 2016 us=356403 Expected Remote Options hash (VER=V4): 'b20ffe30'
Thu Nov 17 14:20:38 2016 us=373683 RESOLVE: Cannot resolve host address: windows-ussouth.cryptostorm.org: The requested name is valid, but no data of the requested type was found.
Thu Nov 17 14:20:38 2016 us=373683 TCP/UDP: Closing socket
Thu Nov 17 14:20:38 2016 us=373683 SIGUSR1[soft,init_instance] received, process restarting
Thu Nov 17 14:20:38 2016 us=373683 Restart pause, 5 second(s)
Thu Nov 17 14:20:43 2016 us=380883 Re-using SSL/TLS context
Thu Nov 17 14:20:43 2016 us=380883 LZO compression initialized
Thu Nov 17 14:20:43 2016 us=380883 Control Channel MTU parms [ L:1604 D:140 EF:40 EB:0 ET:0 EL:0 ]
Thu Nov 17 14:20:43 2016 us=380883 Socket Buffers: R=[8192->8192] S=[8192->8192]
Thu Nov 17 14:20:55 2016 us=398163 RESOLVE: Cannot resolve host address: windows-ussouth.cstorm.pw: The requested name is valid, but no data of the requested type was found.
Thu Nov 17 14:20:55 2016 us=398163 Data Channel MTU parms [ L:1604 D:1450 EF:104 EB:135 ET:0 EL:0 AF:3/1 ]
Thu Nov 17 14:20:55 2016 us=398163 Local Options String: 'V4,dev-type tun,link-mtu 1604,tun-mtu 1500,proto TCPv4_CLIENT,comp-lzo,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-client'
Thu Nov 17 14:20:55 2016 us=398163 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1604,tun-mtu 1500,proto TCPv4_SERVER,comp-lzo,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-server'
Thu Nov 17 14:20:55 2016 us=398163 Local Options hash (VER=V4): '06d8c75c'
Thu Nov 17 14:20:55 2016 us=398163 Expected Remote Options hash (VER=V4): 'b20ffe30'
Thu Nov 17 14:21:07 2016 us=415443 RESOLVE: Cannot resolve host address: windows-ussouth.cstorm.pw: The requested name is valid, but no data of the requested type was found.
Thu Nov 17 14:21:07 2016 us=415443 TCP/UDP: Closing socket
Thu Nov 17 14:21:07 2016 us=415443 SIGUSR1[soft,init_instance] received, process restarting
Thu Nov 17 14:21:07 2016 us=415443 Restart pause, 5 second(s)
Thu Nov 17 14:21:12 2016 us=422643 Re-using SSL/TLS context
Thu Nov 17 14:21:12 2016 us=422643 LZO compression initialized
Thu Nov 17 14:21:12 2016 us=422643 Control Channel MTU parms [ L:1604 D:140 EF:40 EB:0 ET:0 EL:0 ]
Thu Nov 17 14:21:12 2016 us=422643 Socket Buffers: R=[8192->8192] S=[8192->8192]


 ! Message from: parityboy
Added code tags for easier reading


Topic Author
Krin

Re: Serious Drops and packet loss South node

Postby Krin » Fri Nov 18, 2016 10:25 am

To confirm, I would do a trace route to linux-ussouth.cryptostorm.org?

And would this make sense if I was connected? Or should it be done when disconnected to CStorm?

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Fri Nov 18, 2016 11:26 pm

@OP

This part is interesting:

Code: Select all

Thu Nov 17 14:21:07 2016 us=415443 RESOLVE: Cannot resolve host address: windows-ussouth.cstorm.pw: The requested name is valid, but no data of the requested type was found.


It looks like it's trying to re-negotiate the tunnel, but isn't keeping hold of the tunnel's public IP address, and when it does go and do a DNS lookup, it fails - like it receives an empty response perhaps? Or, it's expecting the same address it used last time and is getting different one from the pool. Would that really cause an issue?

While my CS connection is active:
Server:         127.0.0.1
Address: 127.0.0.1#53

Non-authoritative answer:
Name: windows-ussouth.cstorm.pw
Address: 108.62.19.133
Name: windows-ussouth.cstorm.pw
Address: 70.32.38.69


Just out of interest, are you using the Cryptostorm widget or the official OpenVPN client?


Khariz
Posts: 162
Joined: Sun Jan 17, 2016 7:48 am

Re: Serious Drops and packet loss South node

Postby Khariz » Fri Nov 18, 2016 11:34 pm

Yeah, I was just going to mention the same thing. He's not resolving the server after he connects for the first time. That might explain the issue. I don't use the widget by the way. I use OpenVPN on my Windows computer and Viscosity on my Mac.

I love how super clean my logs on are Mac by the way. Windows is so messy:

Code: Select all

Nov 18 12:29:54: Viscosity Mac 1.6.7b5 (1363)
Nov 18 12:29:54: Viscosity OpenVPN Engine Started
Nov 18 12:29:54: Running on Mac OS X 10.12.2
Nov 18 12:29:54: ---------
Nov 18 12:29:54: Checking reachability status of connection...
Nov 18 12:29:54: Connection is reachable. Starting connection attempt.
Nov 18 12:29:56: OpenVPN 2.3.13 x86_64-apple-darwin [SSL (OpenSSL)] [LZO] [PKCS11] [MH] [IPv6] built on Nov  4 2016
Nov 18 12:29:56: library versions: OpenSSL 1.0.2j  26 Sep 2016, LZO 2.09
Nov 18 12:29:59: UDPv4 link local: [undef]
Nov 18 12:29:59: UDPv4 link remote: [AF_INET]108.62.19.132:443
Nov 18 12:29:59: WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Nov 18 12:30:00: [server] Peer Connection Initiated with [AF_INET]108.62.19.132:443
Nov 18 12:30:02: Opening utun (connect(AF_SYS_CONTROL)): Resource busy
Nov 18 12:30:02: Opened utun device utun1
Nov 18 12:30:02: do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Nov 18 12:30:02: /sbin/ifconfig utun1 delete
Nov 18 12:30:02: NOTE: Tried to delete pre-existing tun/tap instance -- No Problem if failure
Nov 18 12:30:02: /sbin/ifconfig utun1 10.33.164.138 10.33.164.138 netmask 255.255.0.0 mtu 1500 up
Nov 18 12:30:02: Initialization Sequence Completed
Nov 18 12:30:02: DNS mode set to: Full


Topic Author
NOYB

Re: Serious Drops and packet loss South node

Postby NOYB » Sat Nov 19, 2016 5:06 am

parityboy wrote:@OP

This part is interesting:

Code: Select all

Thu Nov 17 14:21:07 2016 us=415443 RESOLVE: Cannot resolve host address: windows-ussouth.cstorm.pw: The requested name is valid, but no data of the requested type was found.


It looks like it's trying to re-negotiate the tunnel, but isn't keeping hold of the tunnel's public IP address, and when it does go and do a DNS lookup, it fails - like it receives an empty response perhaps? Or, it's expecting the same address it used last time and is getting different one from the pool. Would that really cause an issue?

While my CS connection is active:
Server:         127.0.0.1
Address: 127.0.0.1#53

Non-authoritative answer:
Name: windows-ussouth.cstorm.pw
Address: 108.62.19.133
Name: windows-ussouth.cstorm.pw
Address: 70.32.38.69


Just out of interest, are you using the Cryptostorm widget or the official OpenVPN client?


Yup, spotted that too. It repeats over and over. Using the Cryptostorm widget "Narwhale".

Now this "feature" may have always been there....but it wasn't an issue until perhaps a month ago. That privoxy vpn proxy has been running for years without drops. Now it drops. And I'd wager that I could drop another 10ea clients into this thing - and they'd all work.

So something changed. And I believe causality has been ruled out here.

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Sat Nov 19, 2016 10:10 pm

@NYOB

Is "Narwhal" the latest client? Might it be worth giving a beta a try if there's one floating around?

EDIT
I decided to capture some logs to see what would happen when my own connection dropped and yup, the same issue. This is on Linux Mint 17.3 KDE.

Code: Select all

Nov 19 22:02:16 vilnius nm-openvpn[7797]: Initialization Sequence Completed
Nov 19 22:46:13 vilnius nm-openvpn[7797]: [server] Inactivity timeout (--ping-restart), restarting
Nov 19 22:46:13 vilnius nm-openvpn[7797]: SIGUSR1[soft,ping-restart] received, process restarting
Nov 19 22:46:15 vilnius nm-openvpn[7797]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Nov 19 22:46:15 vilnius nm-openvpn[7797]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Nov 19 22:46:35 vilnius nm-openvpn[7797]: RESOLVE: Cannot resolve host address: linux-denmark.cryptostorm.net: Temporary failure in name resolution


In this case the connection lasted 44 mins before it fell over while connected to the Denmark node. The part that caught my eye this time around (apart from the failure in DNS resolution) is this:

Code: Select all

Nov 19 22:46:13 vilnius nm-openvpn[7797]: [server] Inactivity timeout (--ping-restart), restarting


Inactivity timeout? Really? So OpenVPN has an issue with an idle connection? I'm sure it never used to - it used to stay up for days.

UPDATE
OK, so when the tunnel gets into this state, the default route is still set to 10.xx.0.1, but that address cannot be pinged. I forgot to check, but it would appear that the openvpn process dies, and leaves the networking configuration in an inconsistent state.

For my own part, I'm going to update my OpenVPN software (currently 2.3.2) to the latest version, and see how it performs.

UPDATE #2
Well, after updating OpenVPN to version 2.3.13, the connection managed to stay up for just shy of 13 hours. See below.

Code: Select all

Nov 20 11:36:49 odessa nm-openvpn[19138]: Initialization Sequence Completed
Nov 21 00:20:59 odessa nm-openvpn[19138]: [server] Inactivity timeout (--ping-restart), restarting
Nov 21 00:20:59 odessa nm-openvpn[19138]: SIGUSR1[soft,ping-restart] received, process restarting
Nov 21 00:21:01 odessa nm-openvpn[19138]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Nov 21 00:21:01 odessa nm-openvpn[19138]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Nov 21 00:21:41 odessa nm-openvpn[19138]: RESOLVE: Cannot resolve host address: linux-england.cryptostorm.net: Temporary failure in name resolution


Topic Author
NOYB

Re: Serious Drops and packet loss South node

Postby NOYB » Mon Nov 21, 2016 8:07 am

parityboy wrote:@NYOB

Is "Narwhal" the latest client? Might it be worth giving a beta a try if there's one floating around?

EDIT
I decided to capture some logs to see what would happen when my own connection dropped and yup, the same issue. This is on Linux Mint 17.3 KDE.

Code: Select all

Nov 19 22:02:16 vilnius nm-openvpn[7797]: Initialization Sequence Completed
Nov 19 22:46:13 vilnius nm-openvpn[7797]: [server] Inactivity timeout (--ping-restart), restarting
Nov 19 22:46:13 vilnius nm-openvpn[7797]: SIGUSR1[soft,ping-restart] received, process restarting
Nov 19 22:46:15 vilnius nm-openvpn[7797]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Nov 19 22:46:15 vilnius nm-openvpn[7797]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Nov 19 22:46:35 vilnius nm-openvpn[7797]: RESOLVE: Cannot resolve host address: linux-denmark.cryptostorm.net: Temporary failure in name resolution


In this case the connection lasted 44 mins before it fell over while connected to the Denmark node. The part that caught my eye this time around (apart from the failure in DNS resolution) is this:

Code: Select all

Nov 19 22:46:13 vilnius nm-openvpn[7797]: [server] Inactivity timeout (--ping-restart), restarting


Inactivity timeout? Really? So OpenVPN has an issue with an idle connection? I'm sure it never used to - it used to stay up for days.

UPDATE
OK, so when the tunnel gets into this state, the default route is still set to 10.xx.0.1, but that address cannot be pinged. I forgot to check, but it would appear that the openvpn process dies, and leaves the networking configuration in an inconsistent state.

For my own part, I'm going to update my OpenVPN software (currently 2.3.2) to the latest version, and see how it performs.

UPDATE #2
Well, after updating OpenVPN to version 2.3.13, the connection managed to stay up for just shy of 13 hours. See below.

Code: Select all

Nov 20 11:36:49 odessa nm-openvpn[19138]: Initialization Sequence Completed
Nov 21 00:20:59 odessa nm-openvpn[19138]: [server] Inactivity timeout (--ping-restart), restarting
Nov 21 00:20:59 odessa nm-openvpn[19138]: SIGUSR1[soft,ping-restart] received, process restarting
Nov 21 00:21:01 odessa nm-openvpn[19138]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Nov 21 00:21:01 odessa nm-openvpn[19138]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Nov 21 00:21:41 odessa nm-openvpn[19138]: RESOLVE: Cannot resolve host address: linux-england.cryptostorm.net: Temporary failure in name resolution


Narwhale is not the latest. I went back to Narwhale because Ver 3.0 seemed to have problems.

I use the Cryptostorm's OpenVPN because it is:

A. Recommended by the vendor (Cryptostorm).
B. Supposedly deals with DNS leakage.

Your comment here is the first solid evidence/clue/help I've received from the "outside world" - but I don't know useful it might be in practice.

Ultimately, I just need Cryptostorm to fix their shit :)

Sometimes use Cryptostorm free when connecting directly from an an Ubuntu box here. Guess I could try that with this token. Ultimately though - I just want the "official" Cryptostorm client fixed.

Should be an embarrassment. Doesn't appear to be.


Topic Author
NOYB

Re: Serious Drops and packet loss South node

Postby NOYB » Mon Nov 21, 2016 8:10 am

parityboy wrote:@NYOB

Is "Narwhal" the latest client? Might it be worth giving a beta a try if there's one floating around?

EDIT
I decided to capture some logs to see what would happen when my own connection dropped and yup, the same issue. This is on Linux Mint 17.3 KDE.




Repeat. "Something changed".

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Mon Nov 21, 2016 3:32 pm

@thread

UPDATE #3
After having updated OpenVPN on two other VMs to version 2.3.13, I can report the following:

On the Italian node, the connection stayed up for 17 mins. On the Danish node the connection stayed up for 1hr 19m. In all cases, "inactivity timeout" seems to be the culprit.

I'm going to experiment with a couple of options and report back.

UPDATE #4
OK, so after adding these options to the config

Code: Select all

ping 15
ping-exit 60

the connection to the Denmark node stayed up for 6hr 45m before again falling to the "inactivity timeout" issue. At this point I'm to leaning towards a bug in the server config.

UPDATE #5
OK, one thing I've noticed with this is that despite the error being labelled "inactivity timeout", the connection will die regardless of any network activity, including a live audio stream. By the way, this is happening on both Linux Mint 17.3 KDE (OpenVPN 2.3.13) and Windows 7 (Widget 2.22/OpenVPN 2.3.6)

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Thu Nov 24, 2016 7:38 pm

@thread

After digging around in another provider's forums, I found a link which may provide a possible solution. This link seems to suggest that the retry values must match on both the client and server configs.

If df or Fermi could pop their heads in and give us some working values for us to test, it would be a good start. :)


Khariz
Posts: 162
Joined: Sun Jan 17, 2016 7:48 am

Re: Serious Drops and packet loss South node

Postby Khariz » Thu Nov 24, 2016 9:14 pm

If you are referring to MTU values, I'm pretty sure that is a wives tale. Another reputable source has said as much. I'll try to find that information and post it.

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Fri Nov 25, 2016 5:06 am

@Khariz

I'm not. It's ping frequency values, not MTU. This issue affects all of my connections on both Linux and Windows. I love CS, but if this isn't resolved soon, I'll have no choice but to go elsewhere. :(


Khariz
Posts: 162
Joined: Sun Jan 17, 2016 7:48 am

Re: Serious Drops and packet loss South node

Postby Khariz » Fri Nov 25, 2016 6:12 am

I'm already primarily using AirVPN. They are the only other guys that seem to take privacy and security seriously. One of the few VPN providers that is OpenVPN only with no other protocols. The Atlanta servers with AirVPN are much more stable than CS South node. It's on sale for 35% off right now too, so it's a good time to try it out.

I'm an aleph token holder over here at CS, so I'll always keep coming back and checking. But I agree that it's not ideal right now.
I've been meaning to mention that I'm now experiencing the same problems as everyone else. I'm technically staying connected to CS but I'm now experiencing the massive packet loss and spikes in latency that others are.

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Fri Nov 25, 2016 6:26 am

@Khariz

I'm starting to wonder...is it a configuration issue on the exit nodes, or are they under some kind of attack? And if it's an attack, why CS and not (for example) AirVPN or PIA?


Khariz
Posts: 162
Joined: Sun Jan 17, 2016 7:48 am

Re: Serious Drops and packet loss South node

Postby Khariz » Fri Nov 25, 2016 6:29 am

Good questions. I haven't been able to resolve since I noticed. I'll post back if I figure anything out. It still works fine for browsing and whatnot, but if I need a super stable connection, I have to swap over to something else.

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Fri Nov 25, 2016 6:42 am

@thread

Well I tried using the IP address of an exit node (as opposed to the FQDN) to see if I could get past the DNS resolution issues - made no difference, apart from the entry in the log.

Code: Select all

Nov 25 01:36:44 vilnius nm-openvpn[2933]: [server] Inactivity timeout (--ping-restart), restarting
Nov 25 01:36:44 vilnius nm-openvpn[2933]: SIGUSR1[soft,ping-restart] received, process restarting
Nov 25 01:36:46 vilnius nm-openvpn[2933]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Nov 25 01:36:46 vilnius nm-openvpn[2933]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Nov 25 01:36:46 vilnius nm-openvpn[2933]: UDPv4 link local: [undef]
Nov 25 01:36:46 vilnius nm-openvpn[2933]: UDPv4 link remote: [AF_INET]176.10.104.50:443
Nov 25 01:36:46 vilnius nm-openvpn[2933]: [server] Peer Connection Initiated with [AF_INET]176.10.104.50:443
Nov 25 01:36:48 vilnius nm-openvpn[2933]: Preserving previous TUN/TAP instance: tun0
Nov 25 01:36:48 vilnius nm-openvpn[2933]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper tun0 1500 1602 10.33.186.74 255.255.0.0 restart
Nov 25 01:36:48 vilnius nm-openvpn[2933]: NOTE: Pulled options changed on restart, will need to close and reopen TUN/TAP device.


Topic Author
NOYB

Re: Serious Drops and packet loss South node

Postby NOYB » Fri Nov 25, 2016 7:09 am

parityboy wrote:@Khariz

I'm starting to wonder...is it a configuration issue on the exit nodes, or are they under some kind of attack? And if it's an attack, why CS and not (for example) AirVPN or PIA?


Never considered the possibility of attack and have no idea how causality could be proven. If that is happening, can see this is very bad news. Might mean people with unlimited budgets could be driving out small businesses. Where does this end?

Perhaps another reason to get Voodoo fully operational if this is indeed going on? Practically, don't understand how that could help either - since the attacker would still know the entry point. Even if it moved around from time to time.

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Fri Nov 25, 2016 7:22 am

NOYB wrote:
Never considered the possibility of attack and have no idea how causality could be proven. If that is happening, can see this is very bad news. Might mean people with unlimited budgets could be driving out small businesses. Where does this end?


Driving the customers of said small businesses into the arms of larger ones, much like herding cattle for the slaughter...


sm1th
Posts: 15
Joined: Wed Sep 07, 2016 1:55 pm

Re: Serious Drops and packet loss South node

Postby sm1th » Fri Nov 25, 2016 1:50 pm

The complete lack of comment from the admins doesn't look good at all.

User avatar

Fermi
Site Admin
Posts: 228
Joined: Tue Jun 17, 2014 11:42 am

Re: Serious Drops and packet loss South node

Postby Fermi » Fri Nov 25, 2016 2:30 pm

I've read this whole thread. It doesn't seem to be restricted to USSouth, is this assumption correct?

Inactivity timeout (--ping-restart), restarting
...
RESOLVE: Cannot resolve host address: xxx.cryptostorm.net: Temporary failure in name resolution


seems to be the common denominator, correct?

ping 20, ping-restart 60 is pushed by the server.

I'm connected the USSouth and have the same issue. I'll connect to another node and see if this issue moves with it.

/fermi

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Fri Nov 25, 2016 7:32 pm

@Fermi

Many thanks for supplying the ping and ping-restart values. :) I'll retest with them as soon as possible. From what I've read, those values must match on both server and client so the obvious question is why aren't they in the client configs?

You are correct in your assumption, and it is also happening on all three platforms, which to me makes it a server-side issue. So far England, Germany, Italy and Switzerland have shown this issue. I haven't fully tested France or Denmark yet, but I have no doubt the issue will manifest itself there as well.


Topic Author
ahnnah_

Re: Serious Drops and packet loss South node

Postby ahnnah_ » Fri Nov 25, 2016 8:00 pm

For the past few days I have been receiving "PID_ERR replay-window backtrack occurred" errors from multiple nodes. Sometimes the connection stays live for a while other times it stays connected but nothing loads after only a few minutes.

User avatar

Fermi
Site Admin
Posts: 228
Joined: Tue Jun 17, 2014 11:42 am

Re: Serious Drops and packet loss South node

Postby Fermi » Fri Nov 25, 2016 9:11 pm

@parityboy,

These values shouldn't be repeated in the client config file I believe. These are pushed by the helper directive --keepalive.

PUSH: Received control message: 'PUSH_REPLY,persist-key,persist-tun,redirect-gateway def1,redirect-gateway bypass-dhcp,dhcp-option DNS 108.62.19.131,route-gateway 10.44.0.1,topology subnet,ping 20,ping-restart 60,ifconfig 10.44.78.120 255.255.0.0'


The TTL of our DNS records is 1337 seconds or 22.3 minutes. If there's an inactivity timeout and the DNS record didn't renew due to ex. key renewal, ping-restart will not work as there's no way to resolve the address anymore.
Increasing the TTL would make sure that the tunnel can be established again, but there will still be a moment where there's no connection right after the inactivity timeout.

So if we can find out what is causing the inactivity timeout this problem will be solved.

Wonder if there's a difference between UDP and TCP connections.

Personally I don't have issues with DE and CH connections.

/fermi

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Sat Nov 26, 2016 6:06 am

@thread

OK, I decided to do one more experiment. My host (running Mint 17.3) was connected to the DE node in the normal way with both the DNS resolver and the IP pushed down from the server. That connection lasted 6h20m.

A VM (also running Mint 17.3 and bridged to the host) was connected to the CH node, but only taking an IP address from the server. The DNS was manually set to a out-of-tunnel on-LAN pfSense DNS forwarder, which in turn is configured to forward to three Cryptostorm DeepDNS resolvers over the naked Internet connection.

So far, that connection has lasted 8 hours. Let's see if it lasts 8 more...

UPDATE
It didn't. :P See below

Code: Select all

Nov 25 16:53:45 vilnius nm-openvpn[5977]: Initialization Sequence Completed
Nov 26 02:06:43 vilnius nm-openvpn[5977]: [server] Inactivity timeout (--ping-restart), restarting
Nov 26 02:06:43 vilnius nm-openvpn[5977]: SIGUSR1[soft,ping-restart] received, process restarting
Nov 26 02:06:45 vilnius nm-openvpn[5977]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Nov 26 02:06:45 vilnius nm-openvpn[5977]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Nov 26 02:07:00 vilnius nm-openvpn[5977]: UDPv4 link local: [undef]
Nov 26 02:07:00 vilnius nm-openvpn[5977]: UDPv4 link remote: [AF_INET]176.10.104.50:443
Nov 26 02:08:00 vilnius nm-openvpn[5977]: [UNDEF] Inactivity timeout (--ping-restart), restarting
Nov 26 02:08:00 vilnius nm-openvpn[5977]: SIGUSR1[soft,ping-restart] received, process restarting
Nov 26 02:08:02 vilnius nm-openvpn[5977]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Nov 26 02:08:02 vilnius nm-openvpn[5977]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Nov 26 02:08:02 vilnius nm-openvpn[5977]: UDPv4 link local: [undef]
Nov 26 02:08:02 vilnius nm-openvpn[5977]: UDPv4 link remote: [AF_INET]185.60.147.79:443
Nov 26 02:08:02 vilnius nm-openvpn[5977]: [server] Peer Connection Initiated with [AF_INET]185.60.147.79:443
Nov 26 02:08:04 vilnius nm-openvpn[5977]: Preserving previous TUN/TAP instance: tun0
Nov 26 02:08:04 vilnius nm-openvpn[5977]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper tun0 1500 1602 10.33.114.185 255.255.0.0 restart
Nov 26 02:08:04 vilnius nm-openvpn[5977]: NOTE: Pulled options changed on restart, will need to close and reopen TUN/TAP device.
Nov 26 02:08:05 vilnius nm-openvpn[5977]: TUN/TAP device tun0 opened
Nov 26 02:08:05 vilnius nm-openvpn[5977]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper tun0 1500 1602 10.33.52.60 255.255.0.0 init
Nov 26 02:08:05 vilnius nm-openvpn[5977]: Initialization Sequence Completed


Looks like it reconnected doesn't it? It did, apart from this

Code: Select all

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
linux-jord1.cry 192.168.1.1     255.255.255.255 UGH   0      0        0 eth0
192.168.1.0     *               255.255.255.0   U     1      0        0 eth0


Ass you can see, the tun0 interface isn't active and the default route hasn't been set. From Network Manager, I got this.

Code: Select all

Nov 26 02:08:04 vilnius nm-openvpn[5977]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper tun0 1500 1602 10.33.114.185 255.255.0.0 restart
Nov 26 02:08:04 vilnius NetworkManager[770]: <info> VPN connection 'CS CH' (IP Config Get) reply received.
Nov 26 02:08:04 vilnius NetworkManager[770]: <error> [1480126084.735340] [nm-vpn-connection.c:695] process_generic_config(): (tun0): failed to look up VPN interface index
Nov 26 02:08:04 vilnius NetworkManager[770]: <info> VPN connection 'CS CH' (IP4 Config Get) reply received.
Nov 26 02:08:04 vilnius NetworkManager[770]:    SCPlugin-Ifupdown: devices removed (path: /sys/devices/virtual/net/tun0, iface: tun0)
Nov 26 02:08:05 vilnius nm-openvpn[5977]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper tun0 1500 1602 10.33.52.60 255.255.0.0 init
Nov 26 02:08:05 vilnius NetworkManager[770]:    SCPlugin-Ifupdown: devices added (path: /sys/devices/virtual/net/tun0, iface: tun0)
Nov 26 02:08:05 vilnius NetworkManager[770]:    SCPlugin-Ifupdown: device added (path: /sys/devices/virtual/net/tun0, iface: tun0): no ifupdown configuration found.
Nov 26 02:08:05 vilnius NetworkManager[770]: <warn> /sys/devices/virtual/net/tun0: couldn't determine device driver; ignoring...
Nov 26 02:08:05 vilnius NetworkManager[770]: <info> VPN connection 'CS CH' (IP Config Get) reply received.
Nov 26 02:08:05 vilnius NetworkManager[770]: <info> VPN connection 'CS CH' (IP4 Config Get) reply received.


@Fermi
Just out of interest, which version of OpenVPN is deployed to the exit nodes?


Topic Author
NOYB

Re: Serious Drops and packet loss South node

Postby NOYB » Sat Nov 26, 2016 8:12 am

parityboy wrote:@thread

OK, I decided to do one more experiment. My host (running Mint 17.3) was connected to the DE node in the normal way with both the DNS resolver and the IP pushed down from the server. That connection lasted 6h20m.

A VM (also running Mint 17.3 and bridged to the host) was connected to the CH node, but only taking an IP address from the server. The DNS was manually set to a out-of-tunnel on-LAN pfSense DNS forwarder, which in turn is configured to forward to three Cryptostorm DeepDNS resolvers over the naked Internet connection.

So far, that connection has lasted 8 hours. Let's see if it lasts 8 more...

UPDATE
It didn't. :P See below

Code: Select all

Nov 25 16:53:45 vilnius nm-openvpn[5977]: Initialization Sequence Completed
Nov 26 02:06:43 vilnius nm-openvpn[5977]: [server] Inactivity timeout (--ping-restart), restarting
Nov 26 02:06:43 vilnius nm-openvpn[5977]: SIGUSR1[soft,ping-restart] received, process restarting
Nov 26 02:06:45 vilnius nm-openvpn[5977]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Nov 26 02:06:45 vilnius nm-openvpn[5977]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Nov 26 02:07:00 vilnius nm-openvpn[5977]: UDPv4 link local: [undef]
Nov 26 02:07:00 vilnius nm-openvpn[5977]: UDPv4 link remote: [AF_INET]176.10.104.50:443
Nov 26 02:08:00 vilnius nm-openvpn[5977]: [UNDEF] Inactivity timeout (--ping-restart), restarting
Nov 26 02:08:00 vilnius nm-openvpn[5977]: SIGUSR1[soft,ping-restart] received, process restarting
Nov 26 02:08:02 vilnius nm-openvpn[5977]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Nov 26 02:08:02 vilnius nm-openvpn[5977]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Nov 26 02:08:02 vilnius nm-openvpn[5977]: UDPv4 link local: [undef]
Nov 26 02:08:02 vilnius nm-openvpn[5977]: UDPv4 link remote: [AF_INET]185.60.147.79:443
Nov 26 02:08:02 vilnius nm-openvpn[5977]: [server] Peer Connection Initiated with [AF_INET]185.60.147.79:443
Nov 26 02:08:04 vilnius nm-openvpn[5977]: Preserving previous TUN/TAP instance: tun0
Nov 26 02:08:04 vilnius nm-openvpn[5977]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper tun0 1500 1602 10.33.114.185 255.255.0.0 restart
Nov 26 02:08:04 vilnius nm-openvpn[5977]: NOTE: Pulled options changed on restart, will need to close and reopen TUN/TAP device.
Nov 26 02:08:05 vilnius nm-openvpn[5977]: TUN/TAP device tun0 opened
Nov 26 02:08:05 vilnius nm-openvpn[5977]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper tun0 1500 1602 10.33.52.60 255.255.0.0 init
Nov 26 02:08:05 vilnius nm-openvpn[5977]: Initialization Sequence Completed


Looks like it reconnected doesn't it? It did, apart from this

Code: Select all

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
linux-jord1.cry 192.168.1.1     255.255.255.255 UGH   0      0        0 eth0
192.168.1.0     *               255.255.255.0   U     1      0        0 eth0


Ass you can see, the tun0 interface isn't active and the default route hasn't been set. From Network Manager, I got this.

Code: Select all

Nov 26 02:08:04 vilnius nm-openvpn[5977]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper tun0 1500 1602 10.33.114.185 255.255.0.0 restart
Nov 26 02:08:04 vilnius NetworkManager[770]: <info> VPN connection 'CS CH' (IP Config Get) reply received.
Nov 26 02:08:04 vilnius NetworkManager[770]: <error> [1480126084.735340] [nm-vpn-connection.c:695] process_generic_config(): (tun0): failed to look up VPN interface index
Nov 26 02:08:04 vilnius NetworkManager[770]: <info> VPN connection 'CS CH' (IP4 Config Get) reply received.
Nov 26 02:08:04 vilnius NetworkManager[770]:    SCPlugin-Ifupdown: devices removed (path: /sys/devices/virtual/net/tun0, iface: tun0)
Nov 26 02:08:05 vilnius nm-openvpn[5977]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper tun0 1500 1602 10.33.52.60 255.255.0.0 init
Nov 26 02:08:05 vilnius NetworkManager[770]:    SCPlugin-Ifupdown: devices added (path: /sys/devices/virtual/net/tun0, iface: tun0)
Nov 26 02:08:05 vilnius NetworkManager[770]:    SCPlugin-Ifupdown: device added (path: /sys/devices/virtual/net/tun0, iface: tun0): no ifupdown configuration found.
Nov 26 02:08:05 vilnius NetworkManager[770]: <warn> /sys/devices/virtual/net/tun0: couldn't determine device driver; ignoring...
Nov 26 02:08:05 vilnius NetworkManager[770]: <info> VPN connection 'CS CH' (IP Config Get) reply received.
Nov 26 02:08:05 vilnius NetworkManager[770]: <info> VPN connection 'CS CH' (IP4 Config Get) reply received.


@Fermi
Just out of interest, which version of OpenVPN is deployed to the exit nodes?



Yeah you've been a trooper, but why are you forced to work this hard?

I'm simply tellin ya, the stable CS client once stayed up for days. Now it doesn't. And no change has been noted for about a month.

The AirVPN client is staying up for exactly 7 days each time (I've got some type of update process hafta find and kill). So the plan now is "buy the 365 day AirVPN" subscription at end of the month. Remove the CS client and place in our software graveyard. (shrug) Not what I wanted. Not anything I ever anticipated.

It's ridiculous you've had to work so hard trying to patch something which never should have been faulty to begin with.

Simple substitution *is* block isolation. Particularly when confirmed by others. The problem is *here*.

Now tell me you really *are* under attack - and I'll stay just to resist. But CS just hasta eat their own gooberisms. VPN service is not supposed to be like this.


Khariz
Posts: 162
Joined: Sun Jan 17, 2016 7:48 am

Re: Serious Drops and packet loss South node

Postby Khariz » Sat Nov 26, 2016 10:49 pm

Buy the AirVPN subscription now. The Black Friday deal is 35% off making a year cost only $37.00.

Doesn't hurt to have either way for that price.

User avatar

cryptostorm_support
ForumHelper
Posts: 296
Joined: Sat Jan 26, 2013 4:31 am
Contact:

Re: Serious Drops and packet loss South node

Postby cryptostorm_support » Sun Nov 27, 2016 12:10 am

@Parityboy

Thanks for your help/input in this. if we can find out what is causing:
[server] Inactivity timeout (--ping-restart), restarting


all subsequent actions causing the trouble will not have to take place.
Adding persist-remote-ip would also help in preventing:
RESOLVE: Cannot resolve host address:


I have sessions on 4 different exit nodes: DE, CH, PT, DK which are stable for days weeks in a row.

@NOYB and others:
We will have a look into this.

/fermi
cryptostorm_support shared support team forum account
PLEASE DON'T SEND PRIVATE MESSAGES with support questions!
--> feel free to use any of our other contact channels, or post in the support forum
cryptostorm: structurally anonymous, token-based, unlimited ☂ bandwidth, opensource, darknet data security for everyone!
keybase.io validatorsonename.io validatorsPGP key @ MITnetwork statuscryptostorm github
support team bitmessage address: BM-2cTMH8K5JnjbfSALjZtSkRWCLfc3Tr8GBV
support team email: support@cryptostorm.is
live chat support: #cryptostorm


Topic Author
NOYB

Re: Serious Drops and packet loss South node

Postby NOYB » Sun Nov 27, 2016 1:34 am

Khariz wrote:Buy the AirVPN subscription now. The Black Friday deal is 35% off making a year cost only $37.00.

Doesn't hurt to have either way for that price.


I thank you for this. My instincts are to avoid Black Friday entirely - so you saved me.


Topic Author
NOYB

Re: Serious Drops and packet loss South node

Postby NOYB » Sun Nov 27, 2016 1:36 am

cryptostorm_support wrote:@Parityboy

Thanks for your help/input in this. if we can find out what is causing:
[server] Inactivity timeout (--ping-restart), restarting


all subsequent actions causing the trouble will not have to take place.
Adding persist-remote-ip would also help in preventing:
RESOLVE: Cannot resolve host address:


I have sessions on 4 different exit nodes: DE, CH, PT, DK which are stable for days weeks in a row.

@NOYB and others:
We will have a look into this.

/fermi


Thanks

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Sun Nov 27, 2016 6:19 am

@Fermi

It might be a good idea to actually check that the server instance is actually sending pings in the first place. If it is, that would point to DPI and/or network interference.

UPDATE
Just to let everyone know, I'm experiencing the very same issue on Android 5.1.1 running Arne Schwab's OpenVPN client, connected to the DE node. This is definitely a server/network issue.


Topic Author
Guest

Re: Serious Drops and packet loss South node

Postby Guest » Mon Nov 28, 2016 1:08 am

parityboy wrote:@Fermi

It might be a good idea to actually check that the server instance is actually sending pings in the first place. If it is, that would point to DPI and/or network interference.

UPDATE
Just to let everyone know, I'm experiencing the very same issue on Android 5.1.1 running Arne Scwhab's OpenVPN client, connected to the DE node. This is definitely a server/network issue.


On Android 6.0 (CM 13) I am getting "Authenticate/Decrypt packet error: bad packet ID (may be a replay)" as well as "Inactivity timeout (--ping-restart)".

User avatar

Fermi
Site Admin
Posts: 228
Joined: Tue Jun 17, 2014 11:42 am

Re: Serious Drops and packet loss South node

Postby Fermi » Mon Nov 28, 2016 3:43 am

@Parityboy,

The "pings" are being sent. These do not behave like ICMP pings. No reply is expected. The client sends ping packets to the server, and the server sends ping packets to the client.
This should be according to the pushed directive ping 20. The client should receive a ping every 20 seconds and the client should send out a ping every 20 seconds.
This is visible when using verb 7 without mute and grepping: SENT PING and RECEIVED PING PACKET

On Windows client we see an inbound ping every 20 seconds, and an outbound ping every 60s. This isn't really according to the OpenVPN man pages?
On Linux client inbound ping (although the same directives are used as the Windows server) isn't every 20 seconds:
22:33:37 2016 us=293779 RECEIVED PING PACKET
22:34:37 2016 us=738332 RECEIVED PING PACKET
22:35:13 2016 us=458883 RECEIVED PING PACKET
22:35:37 2016 us=344539 RECEIVED PING PACKET
22:36:13 2016 us=389301 RECEIVED PING PACKET


same for the ping sent by the client:
21:21:24 2016 us=333098 SENT PING
21:22:19 2016 us=449388 SENT PING
21:22:57 2016 us=334075 SENT PING
21:23:19 2016 us=441406 SENT PING
21:23:57 2016 us=149393 SENT PING


Even if the timing isn't strict, the tunnel stays active as long as pings go back and forth.
If one or both sides have stopped pinging, this will result in a 'Inactivity timeout', followed by a restart after a couple of minutes. (I've seen this situation a couple of times.)

Weird we see different behavior between Linux and Windows, and we see a difference in what is mentioned in the man pages.

df and myself have connections with ussouth that have been stable for hours now (without changes server or client side).

Are you in the possibility to run this verb 7 test?

/fermi


Topic Author
0x24d

Re: Serious Drops and packet loss South node

Postby 0x24d » Mon Nov 28, 2016 4:16 am

Fermi wrote:@Parityboy,

The "pings" are being sent. These do not behave like ICMP pings. No reply is expected. The client sends ping packets to the server, and the server sends ping packets to the client.
This should be according to the pushed directive ping 20. The client should receive a ping every 20 seconds and the client should send out a ping every 20 seconds.
This is visible when using verb 7 without mute and grepping: SENT PING and RECEIVED PING PACKET

On Windows client we see an inbound ping every 20 seconds, and an outbound ping every 60s. This isn't really according to the OpenVPN man pages?
On Linux client inbound ping (although the same directives are used as the Windows server) isn't every 20 seconds:
22:33:37 2016 us=293779 RECEIVED PING PACKET
22:34:37 2016 us=738332 RECEIVED PING PACKET
22:35:13 2016 us=458883 RECEIVED PING PACKET
22:35:37 2016 us=344539 RECEIVED PING PACKET
22:36:13 2016 us=389301 RECEIVED PING PACKET


same for the ping sent by the client:
21:21:24 2016 us=333098 SENT PING
21:22:19 2016 us=449388 SENT PING
21:22:57 2016 us=334075 SENT PING
21:23:19 2016 us=441406 SENT PING
21:23:57 2016 us=149393 SENT PING


Even if the timing isn't strict, the tunnel stays active as long as pings go back and forth.
If one or both sides have stopped pinging, this will result in a 'Inactivity timeout', followed by a restart after a couple of minutes. (I've seen this situation a couple of times.)

Weird we see different behavior between Linux and Windows, and we see a difference in what is mentioned in the man pages.

df and myself have connections with ussouth that have been stable for hours now (without changes server or client side).

Are you in the possibility to run this verb 7 test?

/fermi


I am currently running this test connected to the England node. I will update this post when it is finished.

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Mon Nov 28, 2016 4:52 pm

@Fermi

I'm also currently running this test in a Linux VM while connected to the DK node. As an aside, I've been connected to one of the Cryptofree nodes (on the host) for ~14.5 hours, with no drops. Slow, but reliable. :)

UPDATE
OK so I've been running this test on the DK node for a couple hours now, along with a torrent client which is mostly idle, i.e. there's some continuous network traffic but no upload/download traffic. One thing I've noticed is that even though the server doesn't seem to send PING packets if there's traffic flowing through it, the connection stays active. However, this is at odds with what I've experienced on the England node, where an audio stream was killed by the VPN connection dying.

Also, take a look at this:

Code: Select all

Mon Nov 28 13:47:08 2016 us=995025 SENT PING
Mon Nov 28 13:47:09 2016 us=37716 RECEIVED PING PACKET
Mon Nov 28 14:18:29 2016 us=720566 SENT PING
Mon Nov 28 14:24:02 2016 us=970964 RECEIVED PING PACKET
Mon Nov 28 14:24:04 2016 us=22564 SENT PING
Mon Nov 28 14:25:06 2016 us=417565 SENT PING
Mon Nov 28 14:25:06 2016 us=977230 RECEIVED PING PACKET
Mon Nov 28 14:26:09 2016 us=962394 RECEIVED PING PACKET
Mon Nov 28 14:26:10 2016 us=983903 SENT PING
Mon Nov 28 14:27:14 2016 us=352496 SENT PING
Mon Nov 28 14:27:14 2016 us=392445 RECEIVED PING PACKET


13:47:09 is where I started the torrent client, and 14:18:29 is where I stopped it. However, the client didn't receive a PING from the server until 14:24:02. I will now just let the torrent client do its thing (which is precisely what I do under normal circumstances) and see how long the connection stays up.

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Tue Nov 29, 2016 3:19 am

@thread

OK, here's an excerpt from the DK node.

Code: Select all

Mon Nov 28 16:17:56 2016 us=494804 SENT PING
Mon Nov 28 16:17:56 2016 us=536991 RECEIVED PING PACKET
Mon Nov 28 16:18:16 2016 us=569124 SENT PING
Mon Nov 28 16:18:16 2016 us=611220 RECEIVED PING PACKET
Mon Nov 28 19:59:33 2016 us=32305 RECEIVED PING PACKET
Mon Nov 28 20:00:54 2016 us=278170 SENT PING
Mon Nov 28 20:05:09 2016 us=191954 RECEIVED PING PACKET
Mon Nov 28 20:05:44 2016 us=143904 RECEIVED PING PACKET
Mon Nov 28 20:06:32 2016 us=466662 RECEIVED PING PACKET
Mon Nov 28 20:07:05 2016 us=195165 RECEIVED PING PACKET
Mon Nov 28 20:07:26 2016 us=370024 RECEIVED PING PACKET
Mon Nov 28 20:07:59 2016 us=193517 RECEIVED PING PACKET
Mon Nov 28 20:08:19 2016 us=474228 RECEIVED PING PACKET
Mon Nov 28 20:08:23 2016 us=536906 SENT PING
Mon Nov 28 20:08:39 2016 us=699787 RECEIVED PING PACKET
Mon Nov 28 20:09:05 2016 us=368090 RECEIVED PING PACKET
Mon Nov 28 20:09:26 2016 us=190613 RECEIVED PING PACKET
Mon Nov 28 20:09:26 2016 us=190667 SENT PING
Mon Nov 28 20:09:46 2016 us=190514 RECEIVED PING PACKET
Mon Nov 28 20:10:06 2016 us=378085 RECEIVED PING PACKET
Mon Nov 28 20:10:26 2016 us=669843 RECEIVED PING PACKET
Mon Nov 28 20:10:40 2016 us=35 SENT PING
Mon Nov 28 20:11:05 2016 us=350276 RECEIVED PING PACKET
Mon Nov 28 20:11:13 2016 us=443022 SENT PING
Mon Nov 28 20:11:25 2016 us=187532 RECEIVED PING PACKET
Mon Nov 28 20:11:33 2016 us=409246 SENT PING
Mon Nov 28 20:11:45 2016 us=419354 RECEIVED PING PACKET
Mon Nov 28 20:12:05 2016 us=354623 RECEIVED PING PACKET
Mon Nov 28 20:12:43 2016 us=400417 RECEIVED PING PACKET
Mon Nov 28 20:13:06 2016 us=201927 RECEIVED PING PACKET
Mon Nov 28 20:13:26 2016 us=354683 RECEIVED PING PACKET
Mon Nov 28 20:13:40 2016 us=488660 SENT PING
Mon Nov 28 20:13:46 2016 us=625964 RECEIVED PING PACKET
Mon Nov 28 20:14:06 2016 us=185965 RECEIVED PING PACKET
Mon Nov 28 20:14:27 2016 us=184631 RECEIVED PING PACKET
Mon Nov 28 20:14:41 2016 us=313414 SENT PING
Mon Nov 28 20:15:06 2016 us=391007 RECEIVED PING PACKET
Mon Nov 28 20:15:17 2016 us=643069 SENT PING
Mon Nov 28 20:15:26 2016 us=182592 RECEIVED PING PACKET
Mon Nov 28 20:15:37 2016 us=230909 SENT PING
Mon Nov 28 20:15:46 2016 us=592157 RECEIVED PING PACKET
Mon Nov 28 20:16:07 2016 us=182979 RECEIVED PING PACKET
Mon Nov 28 20:16:27 2016 us=279786 RECEIVED PING PACKET
Mon Nov 28 20:17:06 2016 us=190233 RECEIVED PING PACKET
Mon Nov 28 20:17:27 2016 us=285357 RECEIVED PING PACKET
Mon Nov 28 20:17:40 2016 us=511608 SENT PING
Mon Nov 28 20:17:47 2016 us=192776 RECEIVED PING PACKET
Mon Nov 28 20:18:08 2016 us=189139 RECEIVED PING PACKET
Mon Nov 28 20:18:29 2016 us=284231 RECEIVED PING PACKET
Mon Nov 28 20:25:09 2016 us=336041 RECEIVED PING PACKET
Mon Nov 28 20:25:29 2016 us=518703 RECEIVED PING PACKET
Mon Nov 28 20:25:49 2016 us=570888 RECEIVED PING PACKET
Mon Nov 28 20:26:10 2016 us=218488 RECEIVED PING PACKET
Mon Nov 28 20:26:31 2016 us=285659 RECEIVED PING PACKET
Mon Nov 28 20:26:41 2016 us=446630 SENT PING
Mon Nov 28 20:27:07 2016 us=177574 RECEIVED PING PACKET
Mon Nov 28 20:27:17 2016 us=440991 SENT PING
Mon Nov 28 20:27:28 2016 us=178579 RECEIVED PING PACKET
Mon Nov 28 20:27:39 2016 us=239517 SENT PING
Mon Nov 28 20:27:48 2016 us=624283 RECEIVED PING PACKET
Mon Nov 28 20:28:09 2016 us=71463 RECEIVED PING PACKET
Mon Nov 28 20:28:28 2016 us=196289 RECEIVED PING PACKET
Mon Nov 28 20:28:40 2016 us=219341 SENT PING
Mon Nov 28 20:29:07 2016 us=380558 RECEIVED PING PACKET
Mon Nov 28 20:29:15 2016 us=623985 SENT PING
Mon Nov 28 20:29:28 2016 us=195280 RECEIVED PING PACKET
Mon Nov 28 20:29:39 2016 us=419504 SENT PING
Mon Nov 28 20:29:48 2016 us=953745 RECEIVED PING PACKET
Mon Nov 28 20:30:08 2016 us=526400 RECEIVED PING PACKET
Mon Nov 28 20:30:28 2016 us=614144 RECEIVED PING PACKET
Mon Nov 28 20:30:42 2016 us=919039 SENT PING
Mon Nov 28 20:31:06 2016 us=511007 RECEIVED PING PACKET
Mon Nov 28 20:31:17 2016 us=597899 SENT PING
Mon Nov 28 20:31:26 2016 us=192636 RECEIVED PING PACKET
Mon Nov 28 20:31:40 2016 us=727366 SENT PING
Mon Nov 28 20:31:47 2016 us=15061 RECEIVED PING PACKET
Mon Nov 28 20:32:06 2016 us=457629 RECEIVED PING PACKET
Mon Nov 28 20:32:26 2016 us=888188 RECEIVED PING PACKET
Mon Nov 28 20:32:41 2016 us=253120 SENT PING
Mon Nov 28 20:33:07 2016 us=553513 RECEIVED PING PACKET
Mon Nov 28 20:33:15 2016 us=590322 SENT PING
Mon Nov 28 20:33:27 2016 us=190462 RECEIVED PING PACKET
Mon Nov 28 20:33:40 2016 us=531854 SENT PING
Mon Nov 28 20:33:47 2016 us=583514 RECEIVED PING PACKET
Mon Nov 28 20:34:07 2016 us=959113 RECEIVED PING PACKET
Mon Nov 28 20:34:28 2016 us=103915 RECEIVED PING PACKET
Mon Nov 28 20:34:29 2016 us=260541 SENT PING
Mon Nov 28 20:35:22 2016 us=274804 RECEIVED PING PACKET
Mon Nov 28 20:35:43 2016 us=207933 RECEIVED PING PACKET
Mon Nov 28 20:36:03 2016 us=218688 RECEIVED PING PACKET
Mon Nov 28 20:36:23 2016 us=228152 RECEIVED PING PACKET
Mon Nov 28 20:36:44 2016 us=230765 RECEIVED PING PACKET
Mon Nov 28 20:36:44 2016 us=230819 SENT PING
Mon Nov 28 20:37:32 2016 us=282925 RECEIVED PING PACKET
Mon Nov 28 20:38:26 2016 us=751512 RECEIVED PING PACKET
Mon Nov 28 20:39:20 2016 us=51913 RECEIVED PING PACKET
Mon Nov 28 20:39:30 2016 us=69638 SENT PING
Mon Nov 28 20:39:40 2016 us=62627 RECEIVED PING PACKET
Mon Nov 28 20:40:01 2016 us=97589 RECEIVED PING PACKET
Mon Nov 28 20:40:27 2016 us=127124 RECEIVED PING PACKET
Mon Nov 28 20:40:42 2016 us=449557 SENT PING
Mon Nov 28 20:41:35 2016 us=637697 RECEIVED PING PACKET
Mon Nov 28 20:41:35 2016 us=637750 SENT PING
Mon Nov 28 20:41:56 2016 us=172738 RECEIVED PING PACKET
Mon Nov 28 20:42:16 2016 us=136628 RECEIVED PING PACKET
Mon Nov 28 20:42:36 2016 us=144914 RECEIVED PING PACKET
Mon Nov 28 20:42:42 2016 us=174822 SENT PING
Mon Nov 28 20:42:56 2016 us=179247 RECEIVED PING PACKET
Mon Nov 28 20:43:27 2016 us=534697 SENT PING
Mon Nov 28 20:43:58 2016 us=302850 SENT PING
Mon Nov 28 20:45:08 2016 us=775485 RECEIVED PING PACKET
Mon Nov 28 20:45:35 2016 us=61421 RECEIVED PING PACKET
Mon Nov 28 20:45:41 2016 us=176050 SENT PING
Mon Nov 28 20:45:55 2016 us=443266 RECEIVED PING PACKET
Mon Nov 28 20:46:41 2016 us=348930 RECEIVED PING PACKET
Mon Nov 28 20:46:42 2016 us=538203 SENT PING
Mon Nov 28 20:47:20 2016 us=235589 RECEIVED PING PACKET
Mon Nov 28 20:47:26 2016 us=613281 SENT PING
Mon Nov 28 20:47:40 2016 us=235765 RECEIVED PING PACKET
Mon Nov 28 20:47:46 2016 us=345565 SENT PING
Mon Nov 28 20:48:22 2016 us=269796 RECEIVED PING PACKET
Mon Nov 28 20:54:27 2016 us=944618 RECEIVED PING PACKET
Mon Nov 28 20:55:11 2016 us=416578 RECEIVED PING PACKET
Mon Nov 28 20:55:58 2016 us=238622 RECEIVED PING PACKET
Mon Nov 28 20:56:18 2016 us=699626 RECEIVED PING PACKET
Mon Nov 28 20:56:39 2016 us=425971 RECEIVED PING PACKET
Mon Nov 28 20:57:10 2016 us=251492 RECEIVED PING PACKET
Mon Nov 28 20:57:21 2016 us=654719 SENT PING
Mon Nov 28 20:57:30 2016 us=237951 RECEIVED PING PACKET
Mon Nov 28 20:57:50 2016 us=758986 RECEIVED PING PACKET
Mon Nov 28 20:58:35 2016 us=165138 RECEIVED PING PACKET
Mon Nov 28 20:58:46 2016 us=674423 SENT PING
Mon Nov 28 20:59:34 2016 us=531274 RECEIVED PING PACKET
Mon Nov 28 21:00:21 2016 us=732145 RECEIVED PING PACKET
Mon Nov 28 21:00:42 2016 us=165997 RECEIVED PING PACKET
Mon Nov 28 21:00:43 2016 us=328550 SENT PING
Mon Nov 28 21:01:10 2016 us=285599 RECEIVED PING PACKET
Mon Nov 28 21:01:31 2016 us=375476 RECEIVED PING PACKET
Mon Nov 28 21:01:43 2016 us=421862 SENT PING
Mon Nov 28 21:02:16 2016 us=231588 RECEIVED PING PACKET
Mon Nov 28 21:02:37 2016 us=253311 RECEIVED PING PACKET
Mon Nov 28 21:02:41 2016 us=272914 SENT PING
Mon Nov 28 21:03:09 2016 us=496992 RECEIVED PING PACKET
Mon Nov 28 21:03:53 2016 us=239719 RECEIVED PING PACKET
Mon Nov 28 21:04:27 2016 us=38248 RECEIVED PING PACKET
Mon Nov 28 21:05:34 2016 us=485374 SENT PING
Mon Nov 28 21:05:34 2016 us=527320 RECEIVED PING PACKET
Mon Nov 28 21:05:54 2016 us=758367 RECEIVED PING PACKET
Mon Nov 28 21:06:14 2016 us=216713 RECEIVED PING PACKET
Mon Nov 28 21:06:43 2016 us=99990 RECEIVED PING PACKET
Mon Nov 28 21:06:45 2016 us=353711 SENT PING
Mon Nov 28 21:07:28 2016 us=261176 RECEIVED PING PACKET
Mon Nov 28 21:07:34 2016 us=281132 SENT PING
Mon Nov 28 21:07:48 2016 us=924221 RECEIVED PING PACKET
Mon Nov 28 21:07:54 2016 us=223759 SENT PING
Mon Nov 28 21:08:09 2016 us=230044 RECEIVED PING PACKET
Mon Nov 28 21:08:29 2016 us=497556 RECEIVED PING PACKET
Mon Nov 28 21:09:11 2016 us=170885 RECEIVED PING PACKET
Mon Nov 28 21:09:32 2016 us=183111 RECEIVED PING PACKET
Mon Nov 28 21:09:43 2016 us=427409 SENT PING
Mon Nov 28 21:09:52 2016 us=721502 RECEIVED PING PACKET
Mon Nov 28 21:10:12 2016 us=836587 RECEIVED PING PACKET
Mon Nov 28 21:10:33 2016 us=161466 RECEIVED PING PACKET
Mon Nov 28 21:10:44 2016 us=643182 SENT PING
Mon Nov 28 21:11:11 2016 us=232869 RECEIVED PING PACKET
Mon Nov 28 21:11:19 2016 us=333260 SENT PING
Mon Nov 28 21:11:31 2016 us=449703 RECEIVED PING PACKET
Mon Nov 28 21:11:39 2016 us=726641 SENT PING
Mon Nov 28 21:11:51 2016 us=398428 RECEIVED PING PACKET
Mon Nov 28 21:12:12 2016 us=418228 RECEIVED PING PACKET
Mon Nov 28 21:12:32 2016 us=425654 RECEIVED PING PACKET
Mon Nov 28 21:12:46 2016 us=635514 SENT PING
Mon Nov 28 21:13:11 2016 us=270501 RECEIVED PING PACKET
Mon Nov 28 21:13:21 2016 us=433374 SENT PING
Mon Nov 28 21:13:31 2016 us=424186 RECEIVED PING PACKET
Mon Nov 28 21:13:41 2016 us=875257 SENT PING
Mon Nov 28 21:13:51 2016 us=397590 RECEIVED PING PACKET
Mon Nov 28 21:14:11 2016 us=599784 RECEIVED PING PACKET
Mon Nov 28 21:14:31 2016 us=240666 RECEIVED PING PACKET
Mon Nov 28 21:14:45 2016 us=419596 SENT PING
Mon Nov 28 21:15:12 2016 us=239553 RECEIVED PING PACKET
Mon Nov 28 21:15:19 2016 us=385406 SENT PING
Mon Nov 28 21:15:33 2016 us=428487 RECEIVED PING PACKET
Mon Nov 28 21:15:39 2016 us=724896 SENT PING
Mon Nov 28 21:15:53 2016 us=528377 RECEIVED PING PACKET
Mon Nov 28 21:16:14 2016 us=238858 RECEIVED PING PACKET
Mon Nov 28 21:16:34 2016 us=236771 RECEIVED PING PACKET
Mon Nov 28 21:16:46 2016 us=583848 SENT PING
Mon Nov 28 21:17:11 2016 us=631876 RECEIVED PING PACKET
Mon Nov 28 21:17:22 2016 us=757859 SENT PING
Mon Nov 28 21:17:31 2016 us=944199 RECEIVED PING PACKET
Mon Nov 28 21:17:42 2016 us=167558 SENT PING
Mon Nov 28 21:17:51 2016 us=602786 RECEIVED PING PACKET
Mon Nov 28 21:18:12 2016 us=235198 RECEIVED PING PACKET
Mon Nov 28 21:18:33 2016 us=236047 RECEIVED PING PACKET
Mon Nov 28 21:26:14 2016 us=345776 RECEIVED PING PACKET
Mon Nov 28 21:27:35 2016 us=753152 SENT PING
Mon Nov 28 21:28:15 2016 us=397220 SENT PING
Mon Nov 28 21:28:35 2016 us=580164 SENT PING
Mon Nov 28 21:29:00 2016 us=968284 SENT PING


The gap from 16:18:16 to 19:59:33 is real - no PING packets were sent or received during that time, or at least they were not logged. I'd hazard a guess that it was due to the torrent client keeping the link busy. The VPN connection dropped at around 21:30; over the last two minutes the client sent four PINGs but received none.

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Tue Nov 29, 2016 4:31 pm

@thread

OK, the DK connection has been active for the last 19 hours with no drops. This looks interesting:

Code: Select all

Tue Nov 29 01:04:06 2016 us=85367 RECEIVED PING PACKET
Tue Nov 29 01:04:26 2016 us=571286 RECEIVED PING PACKET
Tue Nov 29 01:04:47 2016 us=194898 RECEIVED PING PACKET
Tue Nov 29 01:06:06 2016 us=114638 RECEIVED PING PACKET
Tue Nov 29 01:06:12 2016 us=197991 SENT PING
Tue Nov 29 01:06:27 2016 us=292332 RECEIVED PING PACKET
Tue Nov 29 04:15:07 2016 us=387888 RECEIVED PING PACKET
Tue Nov 29 04:16:00 2016 us=245109 RECEIVED PING PACKET
Tue Nov 29 04:16:21 2016 us=241663 RECEIVED PING PACKET
Tue Nov 29 04:16:41 2016 us=691503 RECEIVED PING PACKET
Tue Nov 29 04:17:01 2016 us=390713 RECEIVED PING PACKET
Tue Nov 29 04:17:22 2016 us=412375 RECEIVED PING PACKET
Tue Nov 29 04:17:32 2016 us=438420 SENT PING
Tue Nov 29 04:17:57 2016 us=366977 RECEIVED PING PACKET
Tue Nov 29 04:18:08 2016 us=745702 SENT PING
Tue Nov 29 04:18:17 2016 us=736170 RECEIVED PING PACKET


The client doesn't seem to be very active at sending PINGs, under certain circumstances.

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Tue Nov 29, 2016 11:42 pm

@thread

OK, the command line test on the DK node has now lasted 26 hours and it's been stable the entire time, no drops. Hopefully whatever was causing the drops is no longer active. :P


Topic Author
0x24d

Re: Serious Drops and packet loss South node

Postby 0x24d » Wed Nov 30, 2016 12:58 am

parityboy wrote:@thread

OK, the command line test on the DK node has now lasted 26 hours and it's been stable the entire time, no drops. Hopefully whatever was causing the drops is no longer active. :P


I just connected to smoothed -> Chicago and my connection was dropped after 24 minutes. Running the verb 7 test connected to smoothed -> Amsterdam and will see how it goes.


Topic Author
0x24d

Re: Serious Drops and packet loss South node

Postby 0x24d » Wed Nov 30, 2016 1:06 am

Connection to Amsterdam stayed up for a whole 10 minutes: 19:53 - 20:03.

Code: Select all

Tue Nov 29 19:57:26 2016 us=836813 SENT PING
Tue Nov 29 19:58:00 2016 us=884092 SENT PING

Code: Select all

Tue Nov 29 19:57:26 2016 us=927669 RECEIVED PING PACKET
Tue Nov 29 19:58:00 2016 us=884015 RECEIVED PING PACKET


These were the only pings that were sent/received, I was on YouTube like when connected to Chicago.



Khariz
Posts: 162
Joined: Sun Jan 17, 2016 7:48 am

Re: Serious Drops and packet loss South node

Postby Khariz » Wed Nov 30, 2016 1:31 am

It's probably a remnant in an old smoothed.ovpn


Topic Author
0x24d

Re: Serious Drops and packet loss South node

Postby 0x24d » Wed Nov 30, 2016 1:32 am

parityboy wrote:@0x24d

What's the address for the Amsterdam node? I don't see it in the config files on GitHub.


I was connected to 46.165.222.248 but apparently that doesn't exist. The closest node is 46.165.222.246 which is Frankfurt 1.

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Wed Nov 30, 2016 1:46 am

@Khariz

Ahhh, OK.

@0x24d

46.165.222.248 is the Frankfurt instance for Android/iOS/Linux/UNIX. 46.165.222.246 is the DeepDNS resolver on that box, if I remember rightly.


Topic Author
0x24d

Re: Serious Drops and packet loss South node

Postby 0x24d » Wed Nov 30, 2016 1:54 am

parityboy wrote:@Khariz

Ahhh, OK.

@0x24d

46.165.222.248 is the Frankfurt instance for Android/iOS/Linux/UNIX. 46.165.222.246 is the DeepDNS resolver on that box, if I remember rightly.


Ah Okay, I was looking at the IP in the openvpn log, I didn't check the IP whilst connected.


Topic Author
NYOB

Re: Serious Drops and packet loss South node

Postby NYOB » Wed Nov 30, 2016 3:53 am

Lets repeat and be clear about the issue here." In 2016, *any* VPN connection you make to ANY node - should stay up indefinitely.

Resist sugarcoating the problem. "It's gotta work."


Even before the current problems commenced - Cryptostorm connections would usually fail after ~2 weeks. Now given the nature of TCPIP, most people are inclined just to blow such problems off as "uncontrollables". I did. But that's a trap - cuz it keeps you from doing the hard detective work and "overbuilding" it takes to make that connection reliable.

And yeah, if/when that connection fails - there should be zero chance of leakage. Period. Connection should not be possible without a perfectly functioning VPN connection. Arguments otherwise? Just noise.

No one ever claimed running a VPN network would be easy.

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Wed Nov 30, 2016 6:51 am

@thread

OK, second session on the DK node lasted approximately six hours.

UPDATE
The ENG node seems to last 20 mins before it dies.

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Thu Dec 01, 2016 6:45 pm

@Fermi

Just for reference, below is the command line executed by NetworkManager when connecting. This is on a Mint 18 Live CD but the drops are still occurring.

Code: Select all

/usr/sbin/openvpn --remote linux-denmark.cryptostorm.net 443 udp --comp-lzo --nobind --dev tun --cipher AES-256-CBC --auth SHA512 --auth-nocache --reneg-sec 0 --syslog nm-openvpn --mssfix --script-security 2 --up /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper --bus-name org.freedesktop.NetworkManager.openvpn.Connection_30 --tun -- --up-restart --persist-key --persist-tun --management /var/run/NetworkManager/nm-openvpn-4e459987-4c33-4d0c-b1bd-c6246fe297ab unix --management-client-user root --management-client-group root --management-query-passwords --auth-retry interact --route-noexec --ifconfig-noexec --client --auth-user-pass --ca /home/mint/.local/share/networkmanagement/certificates/cstorm_linux_denmark_1-4/ca.crt --user nm-openvpn --group nm-openvpn --chroot /var/lib/openvpn/chroot


See anything odd/wrong?

User avatar

Fermi
Site Admin
Posts: 228
Joined: Tue Jun 17, 2014 11:42 am

Re: Serious Drops and packet loss South node

Postby Fermi » Thu Dec 01, 2016 8:50 pm

seems disturbingly normal ;)

/fermi


Topic Author
0x24d

Re: Serious Drops and packet loss South node

Postby 0x24d » Thu Dec 01, 2016 9:12 pm

I have just started a wired connection to the new Latvia node, I will update this post if/when the connection drops.


Topic Author
0x24d

Re: Serious Drops and packet loss South node

Postby 0x24d » Thu Dec 01, 2016 9:47 pm

Update

Connection to Latvia dropped after 22 minutes.

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Thu Dec 01, 2016 11:30 pm

0x24d wrote:Update

Connection to Latvia dropped after 22 minutes.


This is precisely what I'm seeing on DK, ENG and CH although in the past few days they have stayed up a lot longer in some cases, and died in less than 10 mins in others.

Fermi wrote:seems disturbingly normal ;)
/fermi


lol, I kinda knew you would say that. Couple of things I've noticed:

1) The Cryptofree service seems to stay up without any drops. I'm going to start a test on a Mint 17.3 VM to confirm this.
2) When run from the command line, the OpenVPN client seems to make more effort (or is given a better chance) to recover from "inactivity timeout" errors. When run by Network Manager or the Windows widget, it seems to hang immediately.

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Fri Dec 02, 2016 3:36 am

@thread

OK I've spent the last few hours chatting with Fermi and trying to narrow this issue down. The most recent test was against the England node, where I tested with a 30 second renegotiation interval on Mint 17.3 KDE. The result was

Code: Select all

Thu Dec  1 22:17:31 2016 us=888158 TLS: tls_pre_encrypt: key_id=2
Thu Dec  1 22:17:31 2016 us=888226 UDPv4 WRITE [161] to [AF_INET]5.101.137.252:443: P_DATA_V1 kid=2 DATA len=160
Thu Dec  1 22:17:38 2016 us=130257 UDPv4 WRITE [14] to [AF_INET]5.101.137.252:443: P_CONTROL_SOFT_RESET_V1 kid=3 [ ] pid=0 DATA len=0
Thu Dec  1 22:17:38 2016 us=489534 TUN READ [192]
Thu Dec  1 22:17:38 2016 us=489613 TLS: tls_pre_encrypt: key_id=2
Thu Dec  1 22:17:38 2016 us=489704 UDPv4 WRITE [289] to [AF_INET]5.101.137.252:443: P_DATA_V1 kid=2 DATA len=288
Thu Dec  1 22:17:39 2016 us=656121 TUN READ [60]
Thu Dec  1 22:17:39 2016 us=656199 MSS: 1460 -> 1258
Thu Dec  1 22:17:39 2016 us=656229 TLS: tls_pre_encrypt: key_id=2
Thu Dec  1 22:17:39 2016 us=656331 UDPv4 WRITE [161] to [AF_INET]5.101.137.252:443: P_DATA_V1 kid=2 DATA len=160
Thu Dec  1 22:17:39 2016 us=912124 TUN READ [60]
Thu Dec  1 22:17:39 2016 us=912190 MSS: 1460 -> 1258
Thu Dec  1 22:17:39 2016 us=912217 TLS: tls_pre_encrypt: key_id=2
Thu Dec  1 22:17:39 2016 us=912303 UDPv4 WRITE [161] to [AF_INET]5.101.137.252:443: P_DATA_V1 kid=2 DATA len=160
Thu Dec  1 22:17:52 2016 us=298078 [server] Inactivity timeout (--ping-restart), restarting
Thu Dec  1 22:17:52 2016 us=298168 PID packet_id_free
Thu Dec  1 22:17:52 2016 us=298226 PID packet_id_free
Thu Dec  1 22:17:52 2016 us=298440 PID packet_id_free
Thu Dec  1 22:17:52 2016 us=298487 PID packet_id_free
Thu Dec  1 22:17:52 2016 us=298545 PID packet_id_free
Thu Dec  1 22:17:52 2016 us=298584 PID packet_id_free
Thu Dec  1 22:17:52 2016 us=298613 PID packet_id_free
Thu Dec  1 22:17:52 2016 us=298642 PID packet_id_free
Thu Dec  1 22:17:52 2016 us=298673 TCP/UDP: Closing socket
Thu Dec  1 22:17:52 2016 us=298732 PID packet_id_free
Thu Dec  1 22:17:52 2016 us=298787 SIGUSR1[soft,ping-restart] received, process restarting


In this case (being run from the command line) it recovered for 2 mins before going through the same cycle. When run from Network Manager (or the widget) it will not recover at all. I suspect the SOFT_RESET comes from the client expecting something from the server within a certain time frame and not getting it. If so, this would point to the server getting "stuck" on something during renegotiation and not responding to the client in time.

User avatar

Fermi
Site Admin
Posts: 228
Joined: Tue Jun 17, 2014 11:42 am

Re: Serious Drops and packet loss South node

Postby Fermi » Fri Dec 02, 2016 6:23 pm

London connection still up after 14 hours ...

/fermi

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Fri Dec 02, 2016 9:44 pm

@Fermi

Strange isn't it? For me, every paid Linux instance sited in Europe has displayed this connection error - CH, DE, DK, ENG, FR, IT, LV and PT, none of which can stay up for more than a couple hours on average, sometimes not more than ~20 mins. With accelerated renegotiation intervals (30 - 90 secs), I've been seeing connections last no more than 3-5 mins.

On the other hand, the connection to the Cryptofree node has been solid for the last 21.5 hours, no drops. That's the longest that any CS node has stayed up for since this issue began.

Considering the fact that a) only the paid nodes seem to be affected and b) the free nodes are (likely) in the same DC as the Paris nodes, I'd say the issue is in the configuration of the paid nodes and whatever it is they are doing during renegotiation.

The acid test will be to see if my connection to Cryptofree lasts another 21 hours or so. If so, then (in my opinion) it's definitely a difference in config between the free and paid-for exit nodes.

UPDATE
My connection to PT just died after 3 hours.


Topic Author
NOYB

Re: Serious Drops and packet loss South node

Postby NOYB » Sat Dec 03, 2016 2:51 am

parityboy wrote:@Fermi

On the other hand, the connection to the Cryptofree node has been solid for the last 21.5 hours, no drops. That's the longest that any CS node has stayed up for since this issue began.
UPDATE
My connection to PT just died after 3 hours.


AirVPN - 9 days now

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Sat Dec 03, 2016 5:21 am

@thread

OK, so I tested US East and US South today. Both lasted ~22 mins. My current connection to US North has lasted 6h20m. My current connection to Cryptofree has lasted 29 hours.

UPDATE
lol, well my connection to US North just died. Lasted 6hr24m.

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Sat Dec 03, 2016 5:42 pm

@thread

QUICK UPDATE
Tested the new Russia West node, it lasted ~22 mins before the connection dropped. df is testing a new configuration on the ENG node, so far it has lasted 10 hours with no drops. If it lasts another 10, the new configuration is a step in the right direction.

For a comparison point here's my still-active connection to Cryptofree.

Code: Select all

Dec  1 18:52:23 orsk nm-openvpn[2659]: OpenVPN 2.3.13 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Nov  3 2016
Dec  1 18:52:23 orsk nm-openvpn[2659]: library versions: OpenSSL 1.0.1f 6 Jan 2014, LZO 2.06
Dec  1 18:52:24 orsk nm-openvpn[2659]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Dec  1 18:52:24 orsk nm-openvpn[2659]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Dec  1 18:52:24 orsk nm-openvpn[2659]: UDPv4 link local: [undef]
Dec  1 18:52:24 orsk nm-openvpn[2659]: UDPv4 link remote: [AF_INET]212.129.10.40:443
Dec  1 18:52:25 orsk nm-openvpn[2659]: [server] Peer Connection Initiated with [AF_INET]212.129.10.40:443
Dec  1 18:52:27 orsk nm-openvpn[2659]: TUN/TAP device tun0 opened
Dec  1 18:52:27 orsk nm-openvpn[2659]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper tun0 1500 1602 10.55.0.9 255.255.0.0 init
Dec  1 18:52:27 orsk nm-openvpn[2659]: Initialization Sequence Completed


OK, so that session to Cryptofree had to be terminated; the bandwidth crawled down to nothing. The new session actually lasted 4hr45m before falling victim to "inactivity timeout". That short session is an outlier as far as Cryptofree is concerned though.

In the meanwhile, the session to the ENG node has lasted just shy of 17 hours, while a connection via Widget 2.22 to the Russia West lasted about 23 mins, logs below.

Code: Select all

Sat Dec 03 17:04:29 2016 us=839330 Initialization Sequence Completed
Sat Dec 03 17:24:24 2016 us=547968 VERIFY OK: depth=1, C=CA, ST=QC, L=Montreal, O=Katana Holdings Limite /  cryptostorm_darknet, OU=Tech Ops, CN=cryptostorm_is, emailAddress=certadmin@cryptostorm.is
Sat Dec 03 17:24:24 2016 us=547968 VERIFY OK: nsCertType=SERVER
Sat Dec 03 17:24:24 2016 us=547968 VERIFY OK: depth=0, C=CA, ST=QC, L=Montreal, O=Katana Holdings Limite /  cryptostorm_darknet, OU=Tech Ops, CN=server, emailAddress=certadmin@cryptostorm.is
Sat Dec 03 17:24:25 2016 us=157343 NOTE: --mute triggered...
Sat Dec 03 17:27:07 2016 us=797968 5 variation(s) on previous 3 message(s) suppressed by --mute
Sat Dec 03 17:27:07 2016 us=797968 [server] Inactivity timeout (--ping-restart), restarting
Sat Dec 03 17:27:07 2016 us=860468 TCP/UDP: Closing socket
Sat Dec 03 17:27:07 2016 us=860468 SIGUSR1[soft,ping-restart] received, process restarting
Sat Dec 03 17:27:07 2016 us=860468 Restart pause, 2 second(s)
Sat Dec 03 17:27:09 2016 us=860468 Re-using SSL/TLS context
Sat Dec 03 17:27:09 2016 us=860468 LZO compression initialized
Sat Dec 03 17:27:09 2016 us=860468 Control Channel MTU parms [ L:1606 D:138 EF:38 EB:0 ET:0 EL:0 ]
Sat Dec 03 17:27:09 2016 us=860468 Socket Buffers: R=[8192->8192] S=[8192->8192]
Sat Dec 03 17:27:21 2016 us=860468 RESOLVE: Cannot resolve host address: windows-russiawest.cstorm.pw: The requested name is valid, but no data of the requested type was found.
Sat Dec 03 17:27:21 2016 us=860468 Data Channel MTU parms [ L:1606 D:1450 EF:106 EB:135 ET:0 EL:0 AF:3/1 ]
Sat Dec 03 17:27:21 2016 us=860468 Fragmentation MTU parms [ L:1606 D:1400 EF:105 EB:135 ET:1 EL:0 AF:3/1 ]
Sat Dec 03 17:27:21 2016 us=860468 Local Options String: 'V4,dev-type tun,link-mtu 1606,tun-mtu 1500,proto UDPv4,comp-lzo,mtu-dynamic,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-client'
Sat Dec 03 17:27:21 2016 us=860468 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1606,tun-mtu 1500,proto UDPv4,comp-lzo,mtu-dynamic,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-server'
Sat Dec 03 17:27:21 2016 us=860468 Local Options hash (VER=V4): 'ff724a72'
Sat Dec 03 17:27:21 2016 us=860468 Expected Remote Options hash (VER=V4): '545e2783'


Khariz
Posts: 162
Joined: Sun Jan 17, 2016 7:48 am

Re: Serious Drops and packet loss South node

Postby Khariz » Sun Dec 04, 2016 12:03 am

NOYB wrote:
parityboy wrote:@Fermi

On the other hand, the connection to the Cryptofree node has been solid for the last 21.5 hours, no drops. That's the longest that any CS node has stayed up for since this issue began.
UPDATE
My connection to PT just died after 3 hours.


AirVPN - 9 days now



Yeah, same here:
Albireo United States
United States, Atlanta, Georgia
12d 10h 5m ago

I'd really rather be using CryptoStorm. I get the same good speeds, and feel just as confident security-wise, but I either can't stay connected or my critical connections drop out every 15 minutes or so.

Parityboy has a point. If CryptoFree is saying connected, but the paid CS nodes are not...there has to be a difference in the server configurations. And it's something that "recently" changed, as this was not happening to me a few weeks ago (which is evident from my responses to this very thread).

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Sun Dec 04, 2016 12:40 am

@thread

Log from the Windows Widget 2.22 on the new NL node.

Code: Select all

Sat Dec 03 19:15:56 2016 us=416132 Initialization Sequence Completed
Connected
Sat Dec 03 19:35:48 2016 us=931757 VERIFY OK: depth=1, C=CA, ST=QC, L=Montreal, O=Katana Holdings Limite /  cryptostorm_darknet, OU=Tech Ops, CN=cryptostorm_is, emailAddress=certadmin@cryptostorm.is
Sat Dec 03 19:35:48 2016 us=931757 VERIFY OK: nsCertType=SERVER
Sat Dec 03 19:35:48 2016 us=931757 VERIFY OK: depth=0, C=CA, ST=QC, L=Montreal, O=Katana Holdings Limite /  cryptostorm_darknet, OU=Tech Ops, CN=server, emailAddress=certadmin@cryptostorm.is
Sat Dec 03 19:35:49 2016 us=306757 NOTE: --mute triggered...
Sat Dec 03 19:38:33 2016 us=119257 5 variation(s) on previous 3 message(s) suppressed by --mute
Sat Dec 03 19:38:33 2016 us=119257 [server] Inactivity timeout (--ping-restart), restarting
Sat Dec 03 19:38:33 2016 us=213007 TCP/UDP: Closing socket
Sat Dec 03 19:38:33 2016 us=213007 SIGUSR1[soft,ping-restart] received, process restarting
Sat Dec 03 19:38:33 2016 us=213007 Restart pause, 2 second(s)
Sat Dec 03 19:38:35 2016 us=213007 Re-using SSL/TLS context
Sat Dec 03 19:38:35 2016 us=213007 LZO compression initialized
Sat Dec 03 19:38:35 2016 us=213007 Control Channel MTU parms [ L:1606 D:138 EF:38 EB:0 ET:0 EL:0 ]
Sat Dec 03 19:38:35 2016 us=213007 Socket Buffers: R=[8192->8192] S=[8192->8192]
Sat Dec 03 19:38:47 2016 us=213007 RESOLVE: Cannot resolve host address: windows-netherlands.cstorm.pw: The requested name is valid, but no data of the requested type was found.
Sat Dec 03 19:38:47 2016 us=213007 Data Channel MTU parms [ L:1606 D:1450 EF:106 EB:135 ET:0 EL:0 AF:3/1 ]
Sat Dec 03 19:38:47 2016 us=213007 Fragmentation MTU parms [ L:1606 D:1400 EF:105 EB:135 ET:1 EL:0 AF:3/1 ]
Sat Dec 03 19:38:47 2016 us=213007 Local Options String: 'V4,dev-type tun,link-mtu 1606,tun-mtu 1500,proto UDPv4,comp-lzo,mtu-dynamic,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-client'
Sat Dec 03 19:38:47 2016 us=213007 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1606,tun-mtu 1500,proto UDPv4,comp-lzo,mtu-dynamic,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-server'
Sat Dec 03 19:38:47 2016 us=213007 Local Options hash (VER=V4): 'ff724a72'
Sat Dec 03 19:38:47 2016 us=213007 Expected Remote Options hash (VER=V4): '545e2783'


Died right on time. :P


Khariz
Posts: 162
Joined: Sun Jan 17, 2016 7:48 am

Re: Serious Drops and packet loss South node

Postby Khariz » Sun Dec 04, 2016 1:04 am

Why are you using the 2.22 widget? 3.0.0.64 is the current 3.00 widget. Going to do some tests using that widget right now myself.

Edit: Using the 3.0.0.64 widget. My critical connections dropped out 7 minutes into the attempt. There is nothing in the logs at all. By all accounts in the logs, it's a stable connection, but it's clearly not.

Edit2: Reconnected with my Aleph token instead of my 1 year token, just for giggles, to see if that makes any difference.

Edit3: Lost my connections in only 3 minutes. Oh well. Back to the competitor again.

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Sun Dec 04, 2016 3:19 am

@Khariz

Widget 2.22 uses OpenVPN 2.3.6. I don't know what Widget 3.0.0 uses, but it's obviously newer. On Linux I'm using 2.3.13. What we have proved so far, is that no matter the client platform or OpenVPN version, the problem is reproducible, including on Android 5.1.1/OpenVPN 2.4.x.


Khariz
Posts: 162
Joined: Sun Jan 17, 2016 7:48 am

Re: Serious Drops and packet loss South node

Postby Khariz » Sun Dec 04, 2016 3:27 am

Yep, I've used the 2.4 RC on Windows OpenVPN as well. No dice. Not sure what's going on. Nothing that I can see on my end.

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Sun Dec 04, 2016 4:22 pm

@thread

Well, so far my connections to the NL node have stayed up for around 12 hours, on Windows via the widget and on Linux via Network Manager. Hopefully they'll stay up for another 12. :)

UPDATE
OK, so both of my connections (Linux & Windows) to the NL node have stayed up for 24 hours with no hiccups, so I'm going to declare those connections stable. I'm going to be testing other connections to see what happens, but things look as if they are back to normal so well done to the team for getting to the root of the issue and addressing it (whatever it is you did :P)

UPDATE #2
OK, so the DE, DK, IT, NL, PT and RUS nodes have each been solid for 24 hours, no drops. Since the issue affected all nodes, I believe I've tested enough of them and for long enough to declare the network stable. :thumbup: :D


Topic Author
0x24d

Re: Serious Drops and packet loss South node

Postby 0x24d » Wed Dec 07, 2016 10:14 pm

parityboy wrote:@thread

Well, so far my connections to the NL node have stayed up for around 12 hours, on Windows via the widget and on Linux via Network Manager. Hopefully they'll stay up for another 12. :)

UPDATE
OK, so both of my connections (Linux & Windows) to the NL node have stayed up for 24 hours with no hiccups, so I'm going to declare those connections stable. I'm going to be testing other connections to see what happens, but things look as if they are back to normal so well done to the team for getting to the root of the issue and addressing it (whatever it is you did :P)

UPDATE #2
OK, so the DE, DK, IT, NL, PT and RUS nodes have each been solid for 24 hours, no drops. Since the issue affected all nodes, I believe I've tested enough of them and for long enough to declare the network stable. :thumbup: :D


My connection to Latvia using NetworkManager just dropped after ~20 minutes.

User avatar

parityboy
Site Admin
Posts: 1283
Joined: Wed Feb 05, 2014 3:47 am

Re: Serious Drops and packet loss South node

Postby parityboy » Thu Dec 08, 2016 3:49 am

0x24d wrote:My connection to Latvia using NetworkManager just dropped after ~20 minutes.


My own connections to both Latvia and England have now been solid for 24 hours, not even a hiccup.

  • We are mostly all connected via different networks
  • Many of us are connected to the same nodes
  • Initial authentication is successful, but re-authentication during re-negotiation fails

The last point is somewhat proven by the fact that a) Cryptofree was unaffected and b) for a short while authentication on one of the paid nodes was disabled in order to mimic Cryptofree. Session re-negotiation was successful while authentication was disabled on that node.

Questions
Are our respective ISPs all experimenting with some kind of DPI-based packet mangling which affects (or perhaps even targets) OpenVPN?
Are our ISPs innocent and the fault lies closer to the exit nodes, i.e. in the DC?
If so, why during session re-negotiation and not mid-session?
Why are only some of us affected and not others?

A Proposal For An Experiment
This is aimed at those of you running Linux (since it would be easier to set up) - strange I didn't think of this before. Install a standalone copy of Tor running in client mode, set OpenVPN to run in TCP mode and tell it to use the local Tor relay as a SOCKS proxy. This should rule out any packet interference aimed at OpenVPN.

UPDATE
My connection to LV remains stable, now at nearly 34 hours. The connection to the England node died after 25 hours.


Topic Author
0x24d

Re: Serious Drops and packet loss South node

Postby 0x24d » Sun Dec 11, 2016 3:52 am

I just connected to Moldova and the connection disconnected after 20 minutes (after renegotiation).

I'll try the tor test and see if it is any different.

I am also in a different location/ISP next weekend so I will try connecting and doing the tor test then to see if I get different results.


Topic Author
Guest

Re: Serious Drops and packet loss South node

Postby Guest » Mon Dec 12, 2016 1:45 pm

DD-WRT shows same behaviour, and Manjaro linux will no longer connect at all. Somethings changed. I've been having these probs since right before the thread started- previously rock solid.


Return to “member support & tech assistance”

Who is online

Users browsing this forum: Bing [Bot] and 16 guests

cron

Login