Ξ welcome to cryptostorm's member forums ~ you don't have to be a cryptostorm member to post here Ξ
∞ take a peek at our legendary cryptostorm_is twitter feed if you're into that kind of thing ∞
Ξ we're rolling out voodoo network security across cryptostorm - big things happening, indeed! Ξ
Ξ any OpenVPN configs found on the forum are likely outdated. For the latest, visit GitHub Ξ

Sorry for the disconnect ppl, you'll thank me later :-P

Freewheeling spot to chew the fat on anything cryptostorm-related that doesn't fit elsewhere (i.e. support, howto, &c.). Criticism & praise & brainstorming & requests for explanation... this is where it goes when it's hot & ready for action! :-)
User avatar

Topic Author
df
Site Admin
Posts: 261
Joined: Thu Jan 01, 1970 5:00 am

Sorry for the disconnect ppl, you'll thank me later :-P

Postby df » Fri Aug 28, 2015 10:13 am

I was adding a small feature to the OpenVPN instances server-side, and I noticed a possible vulnerability in something near what I was editing, then ended up playing with it for the past couple of hours to see if it was really vulnerable...

As some of you already know, server-side, our OpenVPN config uses the auth-user-pass-verify directive to call a shell script (auth.sh) that validates the token the client provides against the local mongo database running on that node.

Our auth.sh was small/simple/stable, with very few modifications ever needed over the last couple of years.
At first glance, it seems fine:

Code: Select all

#!/bin/bash
#
TMPFILE="${1}"         # OpenVPN temp file
VPNUSER="`head -n1 ${TMPFILE} | tail -n1`" # hash browns
VPNPASS="`head -n2 ${TMPFILE} | tail -n1`" # super freakin secret paylodes
if [ "$(/sbin/pidof mongod)" == "" ]; then
 /usr/bin/mongod &
fi
# Getting results from Mongo
RESULT=`mongo tokenizer --quiet --eval "db.eval('auth(\'${VPNUSER}\')');"`;
# echo "$RESULT"
# Decision and Logging/Monitoring
if [ "${RESULT}" == "true" ]; then
  echo "Let me in!"
   exit 0
fi
if [ "${RESULT}" == "false" ]; then
  echo "Go away!"
   exit 1
fi


Note: a few weeks ago I switched from via-file to via-env in the server-side configs because sometimes the session up/down scripts would be called after the temporary file OpenVPN created was deleted. The security issues revolving around via-env don't really apply in any sane Linux environment as processes shouldn't be able to read other processes' environment variables, unless they're running under the same UID. And while our OpenVPN instances have almost always ran under the 'nobody' account, no other service does (nginx/Apache/Skifree/etc. get their own account, so a hack of one of those daemons won't compromise OpenVPN).

The older, above script looks a little messy, but the concept is pretty simple, and still applied until the update I made a few minutes ago. Get the token (username) from OpenVPN, do nothing with the password, send the token to the mongo auth() function to see if it's allowed on or if it's expired, then exit 0 if it's allowed, exit 1 otherwise. Also we didn't include an exit 1 at the end of the script like most people would, because if the mongo server goes down for some reason, we would rather people still be able to login (and if a mongo server goes down it's most likely our fault).

Back to the thing I've been wasting the last couple of hours with. I've been trying to get some shell commands executed by providing tokens that contain shell escaping characters like `|$(); etc. but the double + single quotes and the curly braces in the RESULT= line were getting in the way, plus OpenVPN does it's own version of commenting out of some quotes too to make it even more confusing.

But on a test OpenVPN instance, I was able to make $RESULT equal something besides true or false. In one test:

Code: Select all

RESULT='2015-08-28T06:40:20.622+0200 SyntaxError: Unexpected token ILLEGAL'

and in another:

Code: Select all

RESULT='2015-08-28T06:39:00.358+0200 SyntaxError: Unexpected string'

While executing bash commands might not have been possible, it probably was possible to execute arbitrary mongo commands, which might have lead to shell execution maybe through the child_proccess module from node's API.

So the first thing I did was check the mongo logs for any unusual activity, and thankfully, nothing come up. We've got some insanely old mongo dumps on most of the nodes (one was 40gb), but of course nothing client specific was in them.
Just the time/date (from the perspective of that node's timezone) a specific token logged in, and logged out, the rest was a lot of verbose information about what mongo was doing. The login flow is usually: the mongo function auth() to determine whether a provided token is authorized, followed by a session_up() so that the token will expire when it should, and when the client disconnects session_down() is executed.
But I saw nothing in the mongo logs that suggests anyone else has tried or succeeded in exploiting this possible vulnerability.

Before I disclose the code for the current auth.sh and session up/down scripts I just implemented, I should discuss my "small feature" mentioned in the first sentence way above. Some users here on the forum and in other places have commented on the fact that the internal OpenVPN IPs they receive (those 10.* ones) are incremental, so if they connect to a cluster and end up on a specific instance and get an IP of 10.45.0.5 they know there's about 4 other people on the server. This is bad, as it could possibly be used in correlation attacks.

The new feature I added now causes those internal IPs to be randomly generated from within the subnet assigned to that instance.
So instead of 10.45.0.5 the person gets 10.45.95.31 (for example)..

So here's the latest and grateist auth.sh:

Code: Select all

#!/bin/bash
# auth.sh
#
# since it's common for people to accidently leave in a trailing space,
# remove those (and any other non-printable chars) first..
tr -dc '[[:print:]]' <<< "$username"
# then do more input validation on the supplied token...
if grep -P '^[\-a-zA-Z0-9]+$' <<<$username; then
 # token contains only allowed chars, good
 if [[ "${#username}" == 128 ]]; then
  # token is the length of a sha512 hash, good
  hash=$username
 fi
 if [[ "${#username}" == 127 ]]; then
  # fix needed for routers that chop off the last char
  hash=$username
 fi
 if [[ "${#username}" == 126 ]]; then
  # same thing as above
  hash=$username
 fi
 if [[ "${#username}" == 23 ]]; then
  # token is 23 chars long, someone forgot to hash their token,
  # so do it for them, then continue
  hash=`echo -n $username|openssl sha -sha512|awk '{print $NF}'`
 fi
 if [ "$hash" == "" ]; then
  # $hash is empty, which means whatever was provided contained valid chars,
  # but wasn't the length of a token or a hash, so don't let them in
  exit 1
 fi
else
 # token contains invalid chars, possible screwup,
 # possible attempt to inject shell/mongo commands, so reject
 exit 1
fi
# run mongod if not running
if [ "$(/sbin/pidof mongod)" == "" ]; then
 /usr/bin/mongod &
fi
# send the token to mongo auth()
RESULT=`mongo tokenizer --quiet --eval "db.eval('auth(\'${hash}\')');"`;
if [ "${RESULT}" == "true" ]; then
  # good token
  exit 0
fi
if [ "${RESULT}" == "false" ]; then
  # bad/scary/evil/expired token
  exit 1
fi
# no exit 1 here so ppl can still log in if mongo is down


And session_up.sh:

Code: Select all

#!/bin/bash
# up

# tell mongo to session_up() the token
RESULT=`mongo tokenizer --quiet --eval "db.eval('session_up(\'${username}\')');"`;

# figure out the instance name from $config, which contains the full path to the openvpn .conf,
# then use the dir /tmp/[instance name]/pool/ as the dir to hold the files that respresent the
# IPs in the pool that are in use
openvpn_dir=`echo $config|awk -F/ '{print "/tmp/"$3}'`
instance_pool_dir=`echo $openvpn_dir`/pool
POOL=`grep ^server $config|awk '{print $2}'|awk -F. '{print $1"."$2}'`

# create /tmp/[instance name]/pool/ if it's not already there
if [ ! -d $instance_pool_dir ]; then
 mkdir -p $instance_pool_dir
fi

# randomly grab an unused IP from the 10.whatever subnet specific to that instance.
# i.e., for the win udp instance, subnet is 10.44.0.0/16, so assign to the client 10.44.16.58 (for example)
# instead of the default incremental 10.44.0.5 (if there were 4 people on the instance).
found_one=0
while [ "$found_one" -eq "0" ]; do
 # randomly generate the last 2 octets, in the range of 3-254 for each.
 RANDIP=$POOL.`echo $[ 3 + $[ RANDOM % 254 ]]`.`echo $[ 3 + $[ RANDOM % 254 ]]`
 # not taken yet?
 if [ ! -r $instance_pool_dir/$RANDIP ]; then
  # take it
  touch $instance_pool_dir/$RANDIP
  # write out the ifconfig line to $1 which is picked up by openvpn
  echo "ifconfig-push $RANDIP 255.255.0.0" > $1
  found_one=1
 fi
done

# found a random IP and pushed it to the client, so exit happily!
# (or something went horribly wrong, still exit 0 so the client can get in anyways)
exit 0


And the insanely complex session_down.sh:

Code: Select all

#!/bin/bash
# down

# tell mongo to session_down() the token
RESULT=`mongo tokenizer --quiet --eval "db.eval('session_down(\'${username}\')');"`

# figure out what the pool dir is (see session_up.sh)
openvpn_dir=`echo $config|awk -F/ '{print "/tmp/"$3}'`
instance_pool_dir=`echo $openvpn_dir`/pool

# remove their IP from the pool
rm -f $instance_pool_dir/$ifconfig_pool_remote_ip


Note: Keep in mind, the session up/down scripts don't need to repeat the same input validation as auth.sh since auth.sh would prevent the session from ever reaching those scripts if the input validation fails.

Hope that explains things clearly enough :-)

User avatar

marzametal
Posts: 496
Joined: Mon Aug 05, 2013 11:39 am

Re: Sorry for the disconnect ppl, you'll thank me later :-P

Postby marzametal » Fri Aug 28, 2015 10:55 am

During all that lol... something's changed. Could it be possible that the above has changed routing process?

Prior to disconnect, my VPNetMon widget used to display 10.44.0.x or 10.66.0.x when connected to CS.
Post disconnect, it shows stuff like 10.44.9.99 or 10.66.212.112.
Updating firewall rules seems to result in normal CS connection. I jump on ipleak.net and it shows that I am connected to Japan, and DNS address is from Japan as well.

It seems to have gone from the swarm 10.44.0.x to an awfully specific address...

Any clue as to why? I know, Windows. But still... LOL Can't even get on chat!

User avatar

marzametal
Posts: 496
Joined: Mon Aug 05, 2013 11:39 am

Re: Sorry for the disconnect ppl, you'll thank me later :-P

Postby marzametal » Fri Aug 28, 2015 10:58 am

Don't mind me, I just your post again! Phew!


bahhblahblah

Re: Sorry for the disconnect ppl, you'll thank me later :-P

Postby bahhblahblah » Fri Aug 28, 2015 11:08 am

my alph token is still being refused auth network wide.
and irc..
Connecting to cryptostorm.nu (212.83.185.245:6667)
* Connection failed (Connection refused)

remains down.

User avatar

Fermi
ForumHelper
Posts: 194
Joined: Tue Jun 17, 2014 11:42 am

Re: Sorry for the disconnect ppl, you'll thank me later :-P

Postby Fermi » Fri Aug 28, 2015 12:29 pm

marzametal wrote:During all that lol... something's changed. Could it be possible that the above has changed routing process?

Prior to disconnect, my VPNetMon widget used to display 10.44.0.x or 10.66.0.x when connected to CS.
Post disconnect, it shows stuff like 10.44.9.99 or 10.66.212.112.
Updating firewall rules seems to result in normal CS connection. I jump on ipleak.net and it shows that I am connected to Japan, and DNS address is from Japan as well.

It seems to have gone from the swarm 10.44.0.x to an awfully specific address...

Any clue as to why? I know, Windows. But still... LOL Can't even get on chat!


Marzametal,

If you want to chat, you need a temporary workaround ...:
https://cryptostorm.org/viewtopic.php?t=8819&p=14529#p14529

/Fermi

User avatar

Fermi
ForumHelper
Posts: 194
Joined: Tue Jun 17, 2014 11:42 am

Re: Sorry for the disconnect ppl, you'll thank me later :-P

Postby Fermi » Fri Aug 28, 2015 12:31 pm

bahhblahblah wrote:my alph token is still being refused auth network wide.
and irc..
Connecting to cryptostorm.nu (212.83.185.245:6667)
* Connection failed (Connection refused)

remains down.


bahhblahblah,

Please join IRC through .org instead of .nu to sort this out ...
Why? : viewtopic.php?t=8819&p=14529#p14529

/Fermi


linear

Re: Sorry for the disconnect ppl, you'll thank me later :-P

Postby linear » Fri Aug 28, 2015 4:06 pm

Bought token for 1 month on 23.08, all was working and today getting

Code: Select all

AUTH: Received control message: AUTH_FAILED

on all servers. Sitting on free now, but latency is insane.
Is this related?

User avatar

Topic Author
df
Site Admin
Posts: 261
Joined: Thu Jan 01, 1970 5:00 am

Re: Sorry for the disconnect ppl, you'll thank me later :-P

Postby df » Fri Aug 28, 2015 5:39 pm

Sorry for all the auth fails everyone.
Also the IP cryptostorm.is was on was temporarily down due to some abuse complaint nonsense through LeaseWeb.

cryptostorm.is is back up on the original IP, and the auth fails should be fixed now.

Problem was that when adding the input validations to auth.sh, it was also ensuring that only something the length of a token hash (or token) was allowed in (128 bytes). But it turns out, some routers and other devices were actually cutting off the last character of the token hash, but because of a bug (or feature?) in mongo's auth() function, they were allowed in anyways.

So people connecting via routers should be able to get back on now. Added to the input validation a few more lines that also allow token hash lengths of 127 and 126 too since mongo's auth() seems to allow that, but nothing below 126.

Again, sorry for the temporary disconnects due to auth fails, but this was a security fix, so it was better to work past this small bug then to leave a part of the auth system potentially vulnerable.

User avatar

privangle
Posts: 93
Joined: Thu Apr 25, 2013 5:57 am

Re: Sorry for the disconnect ppl, you'll thank me later :-P

Postby privangle » Sat Aug 29, 2015 4:09 am

Thank you for the quick fix of the problem!

Problem was that when adding the input validations to auth.sh, it was also ensuring that only something the length of a token hash (or token) was allowed in (128 bytes). But it turns out, some routers and other devices were actually cutting off the last character of the token hash...

While in bed, I was thinking about that it could be a problem of a number of characters, because the auth_failed response came instantly (which seems to exclude a deep verification calculus in the authentication process). But these "simple" bugs oftenly are not easy to find at all (I say that with my past experience of a coder).

For what I imagine the problem was even harder to find as a "simple" software bug because device depending as you told ("...some routers and other devices..."). So hats to you & technical team! :)

User avatar

Topic Author
df
Site Admin
Posts: 261
Joined: Thu Jan 01, 1970 5:00 am

Re: Sorry for the disconnect ppl, you'll thank me later :-P

Postby df » Sat Aug 29, 2015 5:49 am

Yet Another Update:

After talking with pj & graze, I was reminded that the mongo auth() "bug" is indeed a feature. It was a fix implemented by graze very early on when CS first started.

The problem was that in the OpenVPN source, the file src/openvpn/misc.h uses the code:

Code: Select all

#   define USER_PASS_LEN 128

to set the maximum allowed length for a username or password.

The length of a sha512 hash is exactly 128 characters in length, but in C (the language OpenVPN was written in), the string terminating character (0x00) is added to the end of the string.
So token hashes really ended up being 129 characters in length, which causes the last character or two to be trimmed off the token.

The fix server-side was easy enough, change the 128 in the define to 256.
But it would have been a bad idea to require all clients to recompile their own OpenVPN from source just for this small fix, so graze modified the mongo auth() function to allow for tokens that had the last one or two characters chopped off.

When adding my length checks to the input validation code in auth.sh, I completely forgot about this fix because (to be fair) it was almost 3 years ago.

So that's what happened there :-)

User avatar

marzametal
Posts: 496
Joined: Mon Aug 05, 2013 11:39 am

Re: Sorry for the disconnect ppl, you'll thank me later :-P

Postby marzametal » Sat Aug 29, 2015 1:36 pm

*applauds* well done by all and thanks for showing us customers how things should be done and run!


Return to “general chat, suggestions, industry news”

Who is online

Users browsing this forum: No registered users and 5 guests

Login