Connection once would stay up for days....now dies in minutes. This problem is ongoing - unrelated to any specific time of day.

Noticed one of your hops is labeled "Ubiquiti". It takes extraordinary engineering to yield a reliable microwave link within urban areas using unlicensed spectrum. I hope you haven't been conned.

Fixed it.

Bought AirVPN service and installed on our proxy. No more drops. Kinda suspected that would be the case, because while Frontier Fios is populated by people who mean no good - their service has been pretty reliable.

"Substitution" was the cleanest most concrete way to prove causality. I don't wish to "move" - but what choice are you giving me?

The other nodes are dropping too. While I *expect* to reset our proxy say every two weeks (crytostorm norm)...these every ten minute drops have me scurrying for the exit. Just not good enough.

We use cryptostorm to evade these damn deep packet inspection policies just about every ISP practices...but cryptostorm is proving useless now. Started about 2-3 weeks ago and has never let up.

Thanks for the updates dns resolver listing df and team.

need help with accessibility for RO-RU voodoo node

Same issue. It's been a month or two for me. Connection drops within minutes (usually 5-10 minutes) and won't reconnect. Using Tunnelblick 3.6.8 (build 4625) on Mac OS 10.12.1. Tried North, South, Central. BTW, does anyone actually have specific info on where those nodes are?

Have 11 months remaining on this CS token, perhaps half month remaining on AirVPN.


But AirVPN is still up

Tried again. Stayed up perhaps 10 minutes. Back to AirVPN .....

!	Message from: parityboy
	Added code tags for easier reading

To confirm, I would do a trace route to linux-ussouth.cryptostorm.org?

And would this make sense if I was connected? Or should it be done when disconnected to CStorm?

parityboy wrote:@OP

This part is interesting:

Code: Select all

Thu Nov 17 14:21:07 2016 us=415443 RESOLVE: Cannot resolve host address: windows-ussouth.cstorm.pw: The requested name is valid, but no data of the requested type was found.


It looks like it's trying to re-negotiate the tunnel, but isn't keeping hold of the tunnel's public IP address, and when it does go and do a DNS lookup, it fails - like it receives an empty response perhaps? Or, it's expecting the same address it used last time and is getting different one from the pool. Would that really cause an issue?

While my CS connection is active:

Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
Name:   windows-ussouth.cstorm.pw
Address: 108.62.19.133
Name:   windows-ussouth.cstorm.pw
Address: 70.32.38.69

Just out of interest, are you using the Cryptostorm widget or the official OpenVPN client?

Yup, spotted that too. It repeats over and over. Using the Cryptostorm widget "Narwhale".

Now this "feature" may have always been there....but it wasn't an issue until perhaps a month ago. That privoxy vpn proxy has been running for years without drops. Now it drops. And I'd wager that I could drop another 10ea clients into this thing - and they'd all work.

So something changed. And I believe causality has been ruled out here.

parityboy wrote:@NYOB

Is "Narwhal" the latest client? Might it be worth giving a beta a try if there's one floating around?

EDIT
I decided to capture some logs to see what would happen when my own connection dropped and yup, the same issue. This is on Linux Mint 17.3 KDE.

Code: Select all

Nov 19 22:02:16 vilnius nm-openvpn[7797]: Initialization Sequence Completed
Nov 19 22:46:13 vilnius nm-openvpn[7797]: [server] Inactivity timeout (--ping-restart), restarting
Nov 19 22:46:13 vilnius nm-openvpn[7797]: SIGUSR1[soft,ping-restart] received, process restarting
Nov 19 22:46:15 vilnius nm-openvpn[7797]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Nov 19 22:46:15 vilnius nm-openvpn[7797]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Nov 19 22:46:35 vilnius nm-openvpn[7797]: RESOLVE: Cannot resolve host address: linux-denmark.cryptostorm.net: Temporary failure in name resolution

In this case the connection lasted 44 mins before it fell over while connected to the Denmark node. The part that caught my eye this time around (apart from the failure in DNS resolution) is this:

Code: Select all

Nov 19 22:46:13 vilnius nm-openvpn[7797]: [server] Inactivity timeout (--ping-restart), restarting

Inactivity timeout? Really? So OpenVPN has an issue with an idle connection? I'm sure it never used to - it used to stay up for days.

UPDATE
OK, so when the tunnel gets into this state, the default route is still set to 10.xx.0.1, but that address cannot be pinged. I forgot to check, but it would appear that the openvpn process dies, and leaves the networking configuration in an inconsistent state.

For my own part, I'm going to update my OpenVPN software (currently 2.3.2) to the latest version, and see how it performs.

UPDATE #2
Well, after updating OpenVPN to version 2.3.13, the connection managed to stay up for just shy of 13 hours. See below.

Code: Select all

Nov 20 11:36:49 odessa nm-openvpn[19138]: Initialization Sequence Completed
Nov 21 00:20:59 odessa nm-openvpn[19138]: [server] Inactivity timeout (--ping-restart), restarting
Nov 21 00:20:59 odessa nm-openvpn[19138]: SIGUSR1[soft,ping-restart] received, process restarting
Nov 21 00:21:01 odessa nm-openvpn[19138]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Nov 21 00:21:01 odessa nm-openvpn[19138]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Nov 21 00:21:41 odessa nm-openvpn[19138]: RESOLVE: Cannot resolve host address: linux-england.cryptostorm.net: Temporary failure in name resolution


Narwhale is not the latest. I went back to Narwhale because Ver 3.0 seemed to have problems.

I use the Cryptostorm's OpenVPN because it is:

A. Recommended by the vendor (Cryptostorm).
B. Supposedly deals with DNS leakage.

Your comment here is the first solid evidence/clue/help I've received from the "outside world" - but I don't know useful it might be in practice.

Ultimately, I just need Cryptostorm to fix their shit

Sometimes use Cryptostorm free when connecting directly from an an Ubuntu box here. Guess I could try that with this token. Ultimately though - I just want the "official" Cryptostorm client fixed.

Should be an embarrassment. Doesn't appear to be.

parityboy wrote:@NYOB

Is "Narwhal" the latest client? Might it be worth giving a beta a try if there's one floating around?

EDIT
I decided to capture some logs to see what would happen when my own connection dropped and yup, the same issue. This is on Linux Mint 17.3 KDE.

Repeat. "Something changed".

parityboy wrote:@Khariz

I'm starting to wonder...is it a configuration issue on the exit nodes, or are they under some kind of attack? And if it's an attack, why CS and not (for example) AirVPN or PIA?

Never considered the possibility of attack and have no idea how causality could be proven. If that is happening, can see this is very bad news. Might mean people with unlimited budgets could be driving out small businesses. Where does this end?

Perhaps another reason to get Voodoo fully operational if this is indeed going on? Practically, don't understand how that could help either - since the attacker would still know the entry point. Even if it moved around from time to time.

The complete lack of comment from the admins doesn't look good at all.

I've read this whole thread. It doesn't seem to be restricted to USSouth, is this assumption correct?

Inactivity timeout (--ping-restart), restarting
...
RESOLVE: Cannot resolve host address: xxx.cryptostorm.net: Temporary failure in name resolution

seems to be the common denominator, correct?

ping 20, ping-restart 60 is pushed by the server.

I'm connected the USSouth and have the same issue. I'll connect to another node and see if this issue moves with it.

/fermi

For the past few days I have been receiving "PID_ERR replay-window backtrack occurred" errors from multiple nodes. Sometimes the connection stays live for a while other times it stays connected but nothing loads after only a few minutes.

@parityboy,

These values shouldn't be repeated in the client config file I believe. These are pushed by the helper directive --keepalive.

PUSH: Received control message: 'PUSH_REPLY,persist-key,persist-tun,redirect-gateway def1,redirect-gateway bypass-dhcp,dhcp-option DNS 108.62.19.131,route-gateway 10.44.0.1,topology subnet,ping 20,ping-restart 60,ifconfig 10.44.78.120 255.255.0.0'

The TTL of our DNS records is 1337 seconds or 22.3 minutes. If there's an inactivity timeout and the DNS record didn't renew due to ex. key renewal, ping-restart will not work as there's no way to resolve the address anymore.
Increasing the TTL would make sure that the tunnel can be established again, but there will still be a moment where there's no connection right after the inactivity timeout.

So if we can find out what is causing the inactivity timeout this problem will be solved.

Wonder if there's a difference between UDP and TCP connections.

Personally I don't have issues with DE and CH connections.

/fermi

parityboy wrote:@thread

OK, I decided to do one more experiment. My host (running Mint 17.3) was connected to the DE node in the normal way with both the DNS resolver and the IP pushed down from the server. That connection lasted 6h20m.

A VM (also running Mint 17.3 and bridged to the host) was connected to the CH node, but only taking an IP address from the server. The DNS was manually set to a out-of-tunnel on-LAN pfSense DNS forwarder, which in turn is configured to forward to three Cryptostorm DeepDNS resolvers over the naked Internet connection.

So far, that connection has lasted 8 hours. Let's see if it lasts 8 more...

UPDATE
It didn't. See below

Code: Select all

Nov 25 16:53:45 vilnius nm-openvpn[5977]: Initialization Sequence Completed
Nov 26 02:06:43 vilnius nm-openvpn[5977]: [server] Inactivity timeout (--ping-restart), restarting
Nov 26 02:06:43 vilnius nm-openvpn[5977]: SIGUSR1[soft,ping-restart] received, process restarting
Nov 26 02:06:45 vilnius nm-openvpn[5977]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Nov 26 02:06:45 vilnius nm-openvpn[5977]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Nov 26 02:07:00 vilnius nm-openvpn[5977]: UDPv4 link local: [undef]
Nov 26 02:07:00 vilnius nm-openvpn[5977]: UDPv4 link remote: [AF_INET]176.10.104.50:443
Nov 26 02:08:00 vilnius nm-openvpn[5977]: [UNDEF] Inactivity timeout (--ping-restart), restarting
Nov 26 02:08:00 vilnius nm-openvpn[5977]: SIGUSR1[soft,ping-restart] received, process restarting
Nov 26 02:08:02 vilnius nm-openvpn[5977]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Nov 26 02:08:02 vilnius nm-openvpn[5977]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Nov 26 02:08:02 vilnius nm-openvpn[5977]: UDPv4 link local: [undef]
Nov 26 02:08:02 vilnius nm-openvpn[5977]: UDPv4 link remote: [AF_INET]185.60.147.79:443
Nov 26 02:08:02 vilnius nm-openvpn[5977]: [server] Peer Connection Initiated with [AF_INET]185.60.147.79:443
Nov 26 02:08:04 vilnius nm-openvpn[5977]: Preserving previous TUN/TAP instance: tun0
Nov 26 02:08:04 vilnius nm-openvpn[5977]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper tun0 1500 1602 10.33.114.185 255.255.0.0 restart
Nov 26 02:08:04 vilnius nm-openvpn[5977]: NOTE: Pulled options changed on restart, will need to close and reopen TUN/TAP device.
Nov 26 02:08:05 vilnius nm-openvpn[5977]: TUN/TAP device tun0 opened
Nov 26 02:08:05 vilnius nm-openvpn[5977]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper tun0 1500 1602 10.33.52.60 255.255.0.0 init
Nov 26 02:08:05 vilnius nm-openvpn[5977]: Initialization Sequence Completed


Looks like it reconnected doesn't it? It did, apart from this

Code: Select all

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
linux-jord1.cry 192.168.1.1     255.255.255.255 UGH   0      0        0 eth0
192.168.1.0     *               255.255.255.0   U     1      0        0 eth0


Ass you can see, the tun0 interface isn't active and the default route hasn't been set. From Network Manager, I got this.

Code: Select all

Nov 26 02:08:04 vilnius nm-openvpn[5977]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper tun0 1500 1602 10.33.114.185 255.255.0.0 restart
Nov 26 02:08:04 vilnius NetworkManager[770]: <info> VPN connection 'CS CH' (IP Config Get) reply received.
Nov 26 02:08:04 vilnius NetworkManager[770]: <error> [1480126084.735340] [nm-vpn-connection.c:695] process_generic_config(): (tun0): failed to look up VPN interface index
Nov 26 02:08:04 vilnius NetworkManager[770]: <info> VPN connection 'CS CH' (IP4 Config Get) reply received.
Nov 26 02:08:04 vilnius NetworkManager[770]:    SCPlugin-Ifupdown: devices removed (path: /sys/devices/virtual/net/tun0, iface: tun0)
Nov 26 02:08:05 vilnius nm-openvpn[5977]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper tun0 1500 1602 10.33.52.60 255.255.0.0 init
Nov 26 02:08:05 vilnius NetworkManager[770]:    SCPlugin-Ifupdown: devices added (path: /sys/devices/virtual/net/tun0, iface: tun0)
Nov 26 02:08:05 vilnius NetworkManager[770]:    SCPlugin-Ifupdown: device added (path: /sys/devices/virtual/net/tun0, iface: tun0): no ifupdown configuration found.
Nov 26 02:08:05 vilnius NetworkManager[770]: <warn> /sys/devices/virtual/net/tun0: couldn't determine device driver; ignoring...
Nov 26 02:08:05 vilnius NetworkManager[770]: <info> VPN connection 'CS CH' (IP Config Get) reply received.
Nov 26 02:08:05 vilnius NetworkManager[770]: <info> VPN connection 'CS CH' (IP4 Config Get) reply received.


@Fermi
Just out of interest, which version of OpenVPN is deployed to the exit nodes?

Yeah you've been a trooper, but why are you forced to work this hard?

I'm simply tellin ya, the stable CS client once stayed up for days. Now it doesn't. And no change has been noted for about a month.

The AirVPN client is staying up for exactly 7 days each time (I've got some type of update process hafta find and kill). So the plan now is "buy the 365 day AirVPN" subscription at end of the month. Remove the CS client and place in our software graveyard. (shrug) Not what I wanted. Not anything I ever anticipated.

It's ridiculous you've had to work so hard trying to patch something which never should have been faulty to begin with.

Simple substitution *is* block isolation. Particularly when confirmed by others. The problem is *here*.

Now tell me you really *are* under attack - and I'll stay just to resist. But CS just hasta eat their own gooberisms. VPN service is not supposed to be like this.

@Parityboy

Thanks for your help/input in this. if we can find out what is causing:

[server] Inactivity timeout (--ping-restart), restarting

all subsequent actions causing the trouble will not have to take place.
Adding persist-remote-ip would also help in preventing:

RESOLVE: Cannot resolve host address:

I have sessions on 4 different exit nodes: DE, CH, PT, DK which are stable for days weeks in a row.

@NOYB and others:
We will have a look into this.

/fermi

Khariz wrote:Buy the AirVPN subscription now. The Black Friday deal is 35% off making a year cost only $37.00.

Doesn't hurt to have either way for that price.

I thank you for this. My instincts are to avoid Black Friday entirely - so you saved me.

cryptostorm_support wrote:@Parityboy

Thanks for your help/input in this. if we can find out what is causing:

[server] Inactivity timeout (--ping-restart), restarting

all subsequent actions causing the trouble will not have to take place.
Adding persist-remote-ip would also help in preventing:

RESOLVE: Cannot resolve host address:

I have sessions on 4 different exit nodes: DE, CH, PT, DK which are stable for days weeks in a row.

@NOYB and others:
We will have a look into this.

/fermi

Thanks

parityboy wrote:@Fermi

It might be a good idea to actually check that the server instance is actually sending pings in the first place. If it is, that would point to DPI and/or network interference.

UPDATE
Just to let everyone know, I'm experiencing the very same issue on Android 5.1.1 running Arne Scwhab's OpenVPN client, connected to the DE node. This is definitely a server/network issue.

On Android 6.0 (CM 13) I am getting "Authenticate/Decrypt packet error: bad packet ID (may be a replay)" as well as "Inactivity timeout (--ping-restart)".

@Parityboy,

The "pings" are being sent. These do not behave like ICMP pings. No reply is expected. The client sends ping packets to the server, and the server sends ping packets to the client.
This should be according to the pushed directive ping 20. The client should receive a ping every 20 seconds and the client should send out a ping every 20 seconds.
This is visible when using verb 7 without mute and grepping: SENT PING and RECEIVED PING PACKET

On Windows client we see an inbound ping every 20 seconds, and an outbound ping every 60s. This isn't really according to the OpenVPN man pages?
On Linux client inbound ping (although the same directives are used as the Windows server) isn't every 20 seconds:

22:33:37 2016 us=293779 RECEIVED PING PACKET
22:34:37 2016 us=738332 RECEIVED PING PACKET
22:35:13 2016 us=458883 RECEIVED PING PACKET
22:35:37 2016 us=344539 RECEIVED PING PACKET
22:36:13 2016 us=389301 RECEIVED PING PACKET

same for the ping sent by the client:

21:21:24 2016 us=333098 SENT PING
21:22:19 2016 us=449388 SENT PING
21:22:57 2016 us=334075 SENT PING
21:23:19 2016 us=441406 SENT PING
21:23:57 2016 us=149393 SENT PING

Even if the timing isn't strict, the tunnel stays active as long as pings go back and forth.
If one or both sides have stopped pinging, this will result in a 'Inactivity timeout', followed by a restart after a couple of minutes. (I've seen this situation a couple of times.)

Weird we see different behavior between Linux and Windows, and we see a difference in what is mentioned in the man pages.

df and myself have connections with ussouth that have been stable for hours now (without changes server or client side).

Are you in the possibility to run this verb 7 test?

/fermi

Fermi wrote:@Parityboy,

The "pings" are being sent. These do not behave like ICMP pings. No reply is expected. The client sends ping packets to the server, and the server sends ping packets to the client.
This should be according to the pushed directive ping 20. The client should receive a ping every 20 seconds and the client should send out a ping every 20 seconds.
This is visible when using verb 7 without mute and grepping: SENT PING and RECEIVED PING PACKET

On Windows client we see an inbound ping every 20 seconds, and an outbound ping every 60s. This isn't really according to the OpenVPN man pages?
On Linux client inbound ping (although the same directives are used as the Windows server) isn't every 20 seconds:

22:33:37 2016 us=293779 RECEIVED PING PACKET
22:34:37 2016 us=738332 RECEIVED PING PACKET
22:35:13 2016 us=458883 RECEIVED PING PACKET
22:35:37 2016 us=344539 RECEIVED PING PACKET
22:36:13 2016 us=389301 RECEIVED PING PACKET

same for the ping sent by the client:

21:21:24 2016 us=333098 SENT PING
21:22:19 2016 us=449388 SENT PING
21:22:57 2016 us=334075 SENT PING
21:23:19 2016 us=441406 SENT PING
21:23:57 2016 us=149393 SENT PING

Even if the timing isn't strict, the tunnel stays active as long as pings go back and forth.
If one or both sides have stopped pinging, this will result in a 'Inactivity timeout', followed by a restart after a couple of minutes. (I've seen this situation a couple of times.)

Weird we see different behavior between Linux and Windows, and we see a difference in what is mentioned in the man pages.

df and myself have connections with ussouth that have been stable for hours now (without changes server or client side).

Are you in the possibility to run this verb 7 test?

/fermi

I am currently running this test connected to the England node. I will update this post when it is finished.

parityboy wrote:@thread

OK, the test on the DK node has now lasted 26 hours and it's been stable the entire time, no drops. Hopefully whatever was causing the drops is no longer active.

I just connected to smoothed -> Chicago and my connection was dropped after 24 minutes. Running the verb 7 test connected to smoothed -> Amsterdam and will see how it goes.

Connection to Amsterdam stayed up for a whole 10 minutes: 19:53 - 20:03.




These were the only pings that were sent/received, I was on YouTube like when connected to Chicago.

parityboy wrote:@0x24d

What's the address for the Amsterdam node? I don't see it in the config files on GitHub.

I was connected to 46.165.222.248 but apparently that doesn't exist. The closest node is 46.165.222.246 which is Frankfurt 1.

parityboy wrote:@Khariz

Ahhh, OK.

@0x24d

46.165.222.248 is the Frankfurt instance for Android/iOS/Linux/UNIX. 46.165.222.246 is the DeepDNS resolver on that box, if I remember rightly.

Ah Okay, I was looking at the IP in the openvpn log, I didn't check the IP whilst connected.

Lets repeat and be clear about the issue here." In 2016, *any* VPN connection you make to ANY node - should stay up indefinitely.

Resist sugarcoating the problem. "It's gotta work."


Even before the current problems commenced - Cryptostorm connections would usually fail after ~2 weeks. Now given the nature of TCPIP, most people are inclined just to blow such problems off as "uncontrollables". I did. But that's a trap - cuz it keeps you from doing the hard detective work and "overbuilding" it takes to make that connection reliable.

And yeah, if/when that connection fails - there should be zero chance of leakage. Period. Connection should not be possible without a perfectly functioning VPN connection. Arguments otherwise? Just noise.

No one ever claimed running a VPN network would be easy.

I'm not sure is this related to this exactly but I've noticed when I'm connected to CS (DK NODE) that two of IRC channels always drop out but CS channel doesn't. I've tried in hexchat to use SSL and non but doesn't seem to matter it will drop out like 20 times a day if I keep going.

Furnet servers drop out
Anthrochat servers also drop out

And few times I've noticed when connected to DK node that my speeds just go utterly bad.. like when trying to watch youtube video I'd get 200kbps.. then I disconnect thinking it's my ISP but nope I get solid over 2mpbs download in same video.

seems disturbingly normal

/fermi

I have just started a wired connection to the new Latvia node, I will update this post if/when the connection drops.

Update

Connection to Latvia dropped after 22 minutes.