Ξ welcome to cryptostorm's member forums ~ you don't have to be a cryptostorm member to post here Ξ
∞ take a peek at our legendary cryptostorm_is twitter feed if you're into that kind of thing ∞
Ξ we're rolling out voodoo network security across cryptostorm - big things happening, indeed! Ξ
Ξ any OpenVPN configs found on the forum are likely outdated. For the latest, visit GitHub Ξ

DNS weirdness study guide & mega-thread aka Cthulu

A core mission of cryptostorm is ensuring consistent, reliable network security with minimal fuss & drama. From DNS-based services like our DeepDNS in-browser native .onion/.i2p site access, through grounbreaking research on IP6 leakblocking, & to firewall-based structures to enable "fail-closed" security, this is where we discuss & develop cryptostorm-style leakblock tech.
User avatar

Topic Author
severide
Posts: 27
Joined: Sun Nov 10, 2013 1:09 am

DNS weirdness study guide & mega-thread aka Cthulu

Postby severide » Sat Nov 22, 2014 4:57 am

{direct link: cthulu.cryptostorm.org}


I wanted to bump this thread. I said I was gonna do so about a week or so ago, but I haven't gotten around to it. I'd be interested to see some more discussion on the topic and hope to see guys from the dev team getting involved in this thread.

I had a very brief exchange on twitter about CS firing up their own DNS servers and kinda hounded them about a lack of official response to it. :P

Here's the exchange:
_____________________________
severide ‏@iamseveride Nov 14
@cryptostorm_is you guys should just run your own DNS servers. ez.

_____________________________
cryptostorm°network ‏@cryptostorm_is Nov 14
.@iamseveride - yes,a long running debate in forum
viewtopic.php?f=51&t=6229
viewtopic.php?f=46&t=4186
But no impact wrt local DNS settings...

_____________________________
severide ‏@iamseveride Nov 14
@cryptostorm_is Yeah, I've read both of those threads before. The first thread lacks an official CS response. :p

_____________________________
cryptostorm°network ‏@cryptostorm_is Nov 14
@iamseveride - here goes... Official response: meh, whatever ;-) (in fact, loooong-running internal tech team debate - endless, really)

_____________________________
severide ‏@iamseveride Nov 14
@cryptostorm_is I understand not prioritizing it over other projects (like cryptofree, cleanphone), but makes sense to do it eventually, no?

_____________________________
cryptostorm°network ‏@cryptostorm_is Nov 14
.@iamseveride - tl;dr yes... when we can do it better than current
    1. lots of redundancy
    2. @dnschain?
    3. harden against APT query-spam DoS


One thing that was not mentioned, that I think is pretty important, is peace of mind. I know this is not the usual type of technical discussion that is seen around the forums, but knowing that the DNS servers we're querying belong to and are run by CS provides users with peace of mind. I trust these guys, and sure, there are good solutions out there that are seemingly trustworthy (openNIC, etc), but the team at CS have worked to gain my trust (and others of you out there, I'm sure). I personally would really like to see in-house DNS services. And I also wanted to mention that parityboy has some pretty good suggestions in this thread.

What other ideas do you guys have out there wrt a potential CS-DNS implementation? Do you think something like this is worth a reconsideration from the dev team? I don't think this should take priority over other projects in the pipeline that I'm admittedly much more excited for (stormphone, tunneled tunnels, etc), but I think it's worth it enough to put it on the to-do list!

P.S. Some lengthy post by PJ would not be unwelcome here... :mrgreen:
cryptofree via iOS
CS Node List
CS Wiki maintained by vpnDarknet
PGP
Bitmessage: BM-2cUCkRBnNEhhW3qyNoEpRK6LtQjUs281wT

User avatar

marzametal
Posts: 504
Joined: Mon Aug 05, 2013 11:39 am

Re: [Feature Request] Cryptostorm Dynamic DNS Services

Postby marzametal » Sat Nov 22, 2014 5:44 am

Well, can't make much of a techy comment here... but I have noticed 10.xx.0.16 pop up at least twice.

User avatar

Pattern_Juggled
Posts: 1492
Joined: Sun Dec 16, 2012 6:34 am
Contact:

The Dark One

Postby Pattern_Juggled » Sat Nov 22, 2014 5:43 pm

Cthulu... Dark God Of DNS Weirdness!
cthulhu_by_zaidoigres-d34i3u8.jpg


Right, this is where we'll hash out the stuff that's been going on behind the scenes wrt some... interesting findings in the area of DNS resolution.

Cheers,

    ~ pj

User avatar

vpnDarknet
Posts: 129
Joined: Thu Feb 27, 2014 2:42 pm
Contact:

Re: DNS weirdness study guide & mega-thread aka Cthulu

Postby vpnDarknet » Sun Nov 23, 2014 9:52 am

Long live Cthulu and all is noodly appendage glory, but wouldn't Moorcock's Stormbringer be a more apt daemon for cryptostorm :)


You obviously had a moderately healthy and happy childhood, because if you'd had a conflicted and complex one, you'd know to the very marrow of your bones that Stormbringer is Elric's untrustworthy, soul-sucking demon sword... one of a pair of such cosmic creatures.

Ok, I'll stop now. You can likely already categorise me based on that one sentence alone. :angel:

Cheers,

    ~ pj

ps: besides, Cthulu is well-known for his (its?) mind-warping tendencies... which matches up rather nicely with DNS weirdness.
Buy your tokens via vpnDark.net and cryptostorm cannot and does not know anything about users - no link between a token & purchase details
Unofficial Wiki cryptostorm access guide
Ways to talk to me

User avatar

parityboy
Site Admin
Posts: 1105
Joined: Wed Feb 05, 2014 3:47 am

Re: DNS weirdness study guide & mega-thread aka Cthulu

Postby parityboy » Mon Nov 24, 2014 9:40 pm

@OP

You're absolutely right about the "peace of mind factor" with regard to an in-house DNS service. However, it is important to look at both the purpose of such a service and the usage model(s).

Purpose
The purpose of any DNS service is to translate human-memorable domain names into nework-native IP addresses, basically via a static lookup table - Tor with its g0bbl3d3yg00k.onion domain names is a special case. In all cases, including CS, the DNS service will only be used to resolve clearnet domain names into clearnet IP addresses; having spoken with the techs, CS currently is not architected to support in-darknet services such as websites, due to security concerns. I've been prototyping such a network on my own systems, but I need more time and resources to do a decent job of modelling it properly.

Usage Model(s)
As I wrote above, a DNS service is used to resolve human-memorable domain names into network-sensible IP addresses. Typically, a DNS resolver is provided by the local ISP. You can of course specify a different resolver, such as OpenNIC or OpenDNS, and not use the ISP's one, right? Maybe. Many ISPs intercept port 53 TCP/UDP and redirect it to their own DNS server anyway. Even if they don't do that, DNS traffic is plain text so they can intercept it and read it (and log it). Your ISP's network is a hostile environment, like it or not.

"OK, well let's encrypt it with SSL!" Great idea, except that support for DNS queries over SSL isn't exactly widespread (as far as I know) and even if it was, in an environment where the ISP is forcibly redirecting port 53 TCP/UDP traffic to their own DNS server, the DNS query will (or at least should) fail.

So, how to deal with this? One way is to avoid making any DNS query until the VPN tunnel is active, after which you'll be using a DNS resolver supplied by the OpenVPN server, so all you have to do is connect to your chosen exit node directly using its IP address and ignore the HAF. Great idea, right? Maybe, until the IP address for your exit changes and now you have no VPN until you find out what the new IP address is.

So basically, it's a case of understanding where the peceived threats are (ISP, DNS operator, any point in between), and then limiting the data flowing towards those threat entities. Ideally, what is needed is a setup where your ISP is used to resolve HAF entries, and then once the tunnel is up all further DNS queries are handled by the in-house CS DNS resolver. Your ISP will know at-a-glance that you are using Cryptostorm and may well be curious enough to look deeper, but how much of a threat is that?

User avatar

Pattern_Juggled
Posts: 1492
Joined: Sun Dec 16, 2012 6:34 am
Contact:

in-house resolvers

Postby Pattern_Juggled » Wed Nov 26, 2014 2:46 pm

severide wrote:One thing that was not mentioned, that I think is pretty important, is peace of mind. I know this is not the usual type of technical discussion that is seen around the forums, but knowing that the DNS servers we're querying belong to and are run by CS provides users with peace of mind. I trust these guys, and sure, there are good solutions out there that are seemingly trustworthy (openNIC, etc), but the team at CS have worked to gain my trust (and others of you out there, I'm sure). I personally would really like to see in-house DNS services. And I also wanted to mention that parityboy has some pretty good suggestions in this thread.


There has indeed been excellent discussion of in-house DNS resolvers in several threads (here, and here, and here primarily), over the years. It's been a good process for winnowing down our eventual production-scale approach to the question, and we thank everyone who has participated.

What other ideas do you guys have out there wrt a potential CS-DNS implementation? Do you think something like this is worth a reconsideration from the dev team? I don't think this should take priority over other projects in the pipeline that I'm admittedly much more excited for (stormphone, tunneled tunnels, etc), but I think it's worth it enough to put it on the to-do list!


It's not really been a staff time constraint as much as this question has regulated the process: what can we do in terms of in-house DNS resolution that is substantively better than relying on the well-run and (we feel) reliable outside DNS resolution services we've been using?

Until we could answer that question with confidence, there was not going to be a deployment: doing things in-house just to be able to brag about doing it is bad security procedure: it increases attack surfaces, results in distracting gruntwork taking up staff focus, and requires us to stay abreast of the latest thinking, patches, and needs of this area of tech. If there's a concomitant benefit in terms of member security, we're happy to do so - but without that driver, it's just administrative tech masturbation... fun, sure, but not really all that productive. :-)

Also, DNS stuff is nontrivially complex. This is a truism, but bears repeating. DNS is black magicks, I've joked for years... but it's not a joke. This system is riddled with weird areas of fractal complexity. For those not familiar with this, we encourage them to take a look at the work and (often wonderfully informative as well as entertaining) writing of the inimitable Dan Kaminsky. Dan's super smart, and he understands DNS... all the rest of us are essentially faking it to one degree or another. So when one decides to "do DNS," that's the background: it's not like spinning up an Apache server, and it should not be approached as such.

Finally on this topic, I'd like to point out this strangely informative synchronistic datum: none other than DJB himself coded an alternative (to standard implementations) DNS server: djbDNS. That should tell you something. When you bump into @dakami and @hashbreaker investing considerable personal time in a specific area of tech, you're dealing with nontrivial stuff. That's serious intellectual firepower, as well as enormous functional competence. So, yes, DNS is nontrivial.

(incidentally, DJB's summary of how DNS works is both concise and technically robust; highly recommended)

P.S. Some lengthy post by PJ would not be unwelcome here... :mrgreen:


Ah, yes... my reputation precedes me, I see. ;-)

I could pretty much write forever on DNS and not scratch the surface. But... I consider myself a dilettante in this area - even compared to other areas of tech. I know enough to know what I don't know, and to be fascinated by the layers of knowledge that underlie the system. I also know much of what I might say on such topics is essentially sophomoric and not of much value.

A few months back, we talked with Dan about DNSChain (see below) via twitter. He was kind enough to answer some questions we had about the security model, and the limitations of that model. That's mostly how our team here approaches DNS: when a substantive question comes up, we of course do our own research hand make provisional conclusions. Then we ask smart folks what they think, and sleep on it for a while. Then we repeat. This is true of all serious tech decisions we make, but in the case of DNS even more so.

That said...

Broadly speaking, DNS has a shit-ton of attack surfaces and weak points that present tempting targets for a serious adversary of someone using cryptostorm as a security tool. If you can subvert the process of resolving domain names, as an attacker, then you've got not quite oracular control over their network session but still an enormous leverage over them. These are not theoretical attacks; from simple malware tricks designed to redirect unsuspecting web browsers to adware-laden sites via DNS hijacks of Windows machines, all the way up through Stuxnet and the deep undermining of resolver mechanics that it entailed, DNS is part of so many successful MiTM attacks.

There's oceans of whitepapers and essays and even books on DNS security problems, and proposals to overcome them. If you're curious, dive in! My goal isn't to summarise, as even that's a massive task - but rather to cherry pick the parts that are uniquely relevant to what we do at cryptostorm.

First, DNSSEC doesn't fix the big problems... just in case you were thinking "why not use DNSSEC." Sticking PKI/CA infrastructure on top of a rickey security model simply leaves one with a rickety security model wrapped in a broken identity validation architecture. Not really an improvement.

Second, just running BIND (or whatever) in-house pushes the peas around on the plate but doesn't really eat them. You need to pull down DNS lookup tables from somewhere, of course - in the end you get bound into the existing tiered DNS propagation framework, at some level. Thus going "in-house" isn't tackling the core issues, which have to do with control over those lookup tables and who can assert it. It might be part of a solution, but it's not the solution.

Third, all this is tied up with the laughable - or really tragic is a better word - state of "security" provided by the Certificate Authority (CA) PKI model as currently deployed. I'm not even going to try to summarise why that statement is true. It is, and it's not trivial to "fix" it because the broken current system benefits lots of powerful entities (the NSA, big tech, wealthy registrars with lots of lobbyists on staff, etc.). If you want an intro into the broken-ness, here's @Moxie excellent DefCon presentation, and here's a well-done overview article by the OkTutles folks (the CA stuff is in the middle, mostly). So yes, CA is broken. So that's not really a "solution" to the problem of easily subverted DNS resolution attacks.

Sigh.

Hence we end up at this point: how do you know that the IP address your DNS resolver of choice says is the one appropriate for webresource.whateverTLD when you need to do that mapping locally? This is of course totally relevant to cryptostorm sessions, because cryptostorm sessions begin with a DNS lookup of the HAD "A Record" remote subdomains, in the conf files. An attacker who can redirect those lookups has a toehold on a successful MiTM (only that, and barely that, given that we use purely self-signed CA materials - which you can see why we do that, given the bathetic nature of the PKI model itself - and that spoofing that process is fairly nontrivial even for a well-resourced attacker). And of course most ever network-based resource one uses in daily life involves a DNS lookup: poison the integrity of that process, and chaos reigs.

Hence Cthulu: Dark God of DNS Weirdness :mrgreen:

Our analysis of all this, and close attention paid to what smart folks are saying who really, really understand this stuff, has lead us to conclude that Namecoin-style DNS resolution systems are the meta-solution to the deep structural problems outlined above. To explain why gets into blockchain theory and whatnot, and there's plenty of good writing on that out there already. Namecoin itself as a DNS "service" is a bit... proof-of-concept-ish.

But then there's DNSChain - it's blockchain-based, it's namecoin-derived, and it's not totally decoupled from the legacy/centralised DNS model as it currently exists. Hence we've been following their development closely, and asking smart people about how they see the project.

In the last month or so, our conclusion has been that a beta deployment is appropriate. Hence that's what we're doing.

Phew.

Wordy enough, severide?

Cheers,

    ~ pj

User avatar

Topic Author
severide
Posts: 27
Joined: Sun Nov 10, 2013 1:09 am

Re: in-house resolvers

Postby severide » Thu Nov 27, 2014 7:26 am

Thanks a lot for this reply, pj!

Pattern_Juggled wrote:doing things in-house just to be able to brag about doing it is bad security procedure: it increases attack surfaces, results in distracting gruntwork taking up staff focus, and requires us to stay abreast of the latest thinking, patches, and needs of this area of tech.


I did take "staff focus" into account when mentioning project priority. But, I hadn't put much thought into increased attack surfaces. I suppose this is akin to "all eggs in one basket" ? Would cs users be just as vulnerable to DNS attack vectors as those not connected via the network?

From parityboy's post:
parityboy wrote:Many ISPs intercept port 53 TCP/UDP and redirect it to their own DNS server anyway. Even if they don't do that, DNS traffic is plain text so [your ISP] can intercept it and read it (and log it)


Isn't this true only for those not connected to CS, etc? I suppose my assumption of the in-house DNS resolver was to be used in conjunction with cs tunnel. Would this still be the case if DNSCurve (or equivalent) was implemented? Can it work with a DNSChain setup? Or is dnscurve meant for traditional DNS resolvers? Sorry if this is obvious, but I must admit to being a DNS n00b. I know enough to understand the very basics. :P



Pattern_Juggled wrote:(incidentally, DJB's summary of how DNS works is both concise and technically robust; highly recommended)


Hadn't seen this write-up before. Concise, indeed. Nice read.


Pattern_Juggled wrote:If you want an intro into the broken-ness, here's @Moxie excellent DefCon presentation


I remember watching that a couple years back. Great presentation, and his convergence idea was/is great. It's too bad that it didn't take off. Either he lost interest in it or it's because other projects took up his time (RedPhone, TextSecure, etc).

Pattern_Juggled wrote:But then there's DNSChain - it's blockchain-based, it's namecoin-derived, and it's not totally decoupled from the legacy/centralised DNS model as it currently exists. Hence we've been following their development closely, and asking smart people about how they see the project. In the last month or so, our conclusion has been that a beta deployment is appropriate. Hence that's what we're doing.


This is pretty exciting stuff! DNSChain is really interesting to me. Looking forward to being a guinea pig. :D


Pattern_Juggled wrote:Phew.

Wordy enough, severide?


Yeah, that'll do. Heh. :mrgreen:
cryptofree via iOS
CS Node List
CS Wiki maintained by vpnDarknet
PGP
Bitmessage: BM-2cUCkRBnNEhhW3qyNoEpRK6LtQjUs281wT

User avatar

parityboy
Site Admin
Posts: 1105
Joined: Wed Feb 05, 2014 3:47 am

Re: DNS weirdness study guide & mega-thread aka Cthulu

Postby parityboy » Thu Nov 27, 2014 2:44 pm

@severide

Isn't this true only for those not connected to CS, etc? I suppose my assumption of the in-house DNS resolver was to be used in conjunction with cs tunnel.


"Those not connected to CS" also includes those CS users who a) don't have their tunnel active yet and b) use the domain name of their chosen exit, rather than the IP address. This most certainly includes all Windows widget users, as well as many Linux and OS X users. raw-onyx-1.cryptostorm.net has to resolve to an IP address, and this is before the tunnel is active.

Once the tunnel is active, the client machine should be updated with what ever DNS resolver is supplied by the exit node, and then when the tunnel drops the client should revert back to the default DNS resolver, what that might be.

User avatar

Pattern_Juggled
Posts: 1492
Joined: Sun Dec 16, 2012 6:34 am
Contact:

Re: DNS weirdness study guide & mega-thread aka Cthulu

Postby Pattern_Juggled » Fri Nov 28, 2014 11:35 am

parityboy wrote:"Those not connected to CS" also includes those CS users who a) don't have their tunnel active yet and b) use the domain name of their chosen exit, rather than the IP address. This most certainly includes all Windows widget users, as well as many Linux and OS X users. raw-onyx-1.cryptostorm.net has to resolve to an IP address, and this is before the tunnel is active.


This is where the difficult questions start.

There is, in our current model, a step prior to session instantiation that requires a DNS lookup for the node and instance IP. That step helps enormously when it comes to session resilience and durability against attacks on individual nodes. It is also an obvious attack point itself, for if you can block the HAF lookups then you can block the session connection to cryptostorm in the first place.

And, of course, you can't tunnel through the tunnel the lookups required to create the tunnel. Seems obvious, but any of us who has done this a while has inadvertently set that up, and had it cycle in an infinite regressive loop.

For this pre-tunnel step, do we make effort to direct those DNS queries towards our in-house framework for DNS lookups? There's a good argument for that... and if we do, why not get clever and do some sort of transient tunnelling (via, say, an ICMP pingtunnel) to shield those queries from overtly malicious fiddling? To me, anyhow, that makes sense. It also opens the door to getting clever with other session-initiation traffic, which currently stands out like a beacon on a DPI-layer inspection of the local segment. This moves you away from a naive deployment of openvpn, and more towards forked versions... or towards wrappers/libraries sitting on top of openvpn. In any case, it gets interesting.

Once the tunnel is active, the client machine should be updated with what ever DNS resolver is supplied by the exit node, and then when the tunnel drops the client should revert back to the default DNS resolver, what that might be.


Yes, in-session DNS resolution is pushed down during session setup; we will change those settings (which can be seen as published here in serverconf.cryptostorm.org) to our in-house alternatives once they are ready for use. This is, actually, kind of a hand-wave process because we could also redirect those queries, server-side, to DNS resolvers as they come out of the tunnel; we don't do that, as a matter of philosophical stance... but technically it is more or less commutative with pushed DNS resolvers during session setup.

In the end, the question of how far into the DNS settings of client machines cryptostorm seeks to insert itself is the big one to ask: to "solve" most any form of DNS "leaks," particularly IP6 leaks (see below post), requires a somewhat aggressive position on this issue. Unfortunately, once one begins fiddling with these settings client-side, the risk of OS weirdness goes up as well... and that is a responsibility we take seriously.

Cheers,

    ~ pj

User avatar

Pattern_Juggled
Posts: 1492
Joined: Sun Dec 16, 2012 6:34 am
Contact:

"DNS leak test" websites & IP6 leaks

Postby Pattern_Juggled » Fri Nov 28, 2014 1:00 pm

Ok, this is a very loosely-structured post as we're asking for member community assistance and despite a few weeks of fairly intensive work in-house, we still don't have things boiled down to a good, strong summary. I've begun work on this post several times, and had other tech team folks contribute both bits of writing and general conceptual guidance to the document... only to throw it out and start from scratch repeatedly.

The problem starts like this:

A few weeks back, members asked us to take a look at results being produced by the website dnskeaktest.com, which is a nice little nonprofit service created by a CS student to help folks check their network sessions for "leaks." Oddly, some members on fully-verified (via other means) cryptostorm sessions were getting dns resolvers showing up in the results when they visited that site. Some members... but far from all of them. Even within the same OS, it was hit-and-miss.

This is the kind of issue that tends to lead to important questions, and for a couple of months our support folks have had an ear to ground for any such reports. When we get them, we ask members to tell us all they can about their local machine, OS, local network, etc. Without a theory to run with, this was a bit of a wild goose chase. Many members have been patient and helpful through this, and we offer our thanks (especially @shrouded, who did extensive testing that lead to our current understanding). As the weeks and then a few months wore on, the issue escalated in our tech team; we'd come at it from different angles, frustrated we couldn't really pin down specific theories to test.

Now, I believe we know what's going on... and this post is, in part, a request for folks to help verify our provisional findings. If this holds up, we already have an approach to it that brings things up a notch in terms of session security... but until we get this validated, we've hesitated to start pushing out "solutions" to a problem we don't fully understand.

Let me step back and hit that point squarely:

"DNS leaks" are a hot topic, when it comes to "VPN services" and especially to technically unsophisticated reviewers. Every me-too "VPN services" has a "DNS leak blocker" they claim magically solves this issue. Unfortunately, there's a few problems here. The definition of "DNS leak" ends up being entirely opaque, leading one to wonder how one can claim to "solve" a problem that cannot even be described properly! I've written about that issue in a long essay you can find here, if you're curious. Worse, these "DNS leak" prevention tools being pedalled by just about every "VPN service" out there are basically junk. This I know because I've sat and analysed enough pcap files via wireshark (another pending post, still in pre-publication edit for now) to see the leaks firsthand, via actual packets leaving actual NICs on actual computers. Those results often don't jibe with the results presented by "DNS leak" detection websites. More on that, below.

Indeed, some of the tools being pedalled by "VPN services" are abandonware that legitimate DNS projects have stepped back from and no longer support. So why do they keep getting pushed? One, it looks good to claim a "magical answer" to the problem: the review sites check off that box, never bothering to see if they actually work. Marketing hype, basically. Two, some of these tools succeed in making the DNS leak websites show no leak... even though the leaks are still taking place in actual network sessions, as verified by pcaps.

So, we've not pushed out some "solution" to this issue until we fully understood it. And our research, much of what I've done in bits and pieces over the summer months, keeps running up against anomalies, mutually conflicting explanations, hand-waving, and just plain junk. This has slowed things down, as I'd simply hoped to find folks really smart in this, and ask them what's going on: that's a great approach, and one we prefer around here. There's plenty of smart DNS folks, often generous in their time and expertise, but this "DNS leak" question is so poorly specified and so poorly framed that they generally recoil in disgust when asked about it. And: fair enough. I get it. It's like being asked if internal combustion engines are fast or slow... a vague, poorly-defined, frustrating question.

Back at the core of things:

As I poked at this question, and asked smart folks to help me see why the question itself is so wobbly, I noticed something obvious - in hindsight. We're talking about "DNS leaks" and yet... to "test" them we visit websites. Hold on a second. Sure yes, in general the website does what? It asks the client machine: "hey I want to have you look up an IP address for a domain name... what DNS resolver will you use for that?" Which makes sense but also is a bit wtf. Why should a webserver sitting halfway 'round the world be able to ask a client PC to do DNS resolutions for it? That doesn't make any sense at all.

Since I'm slow on the update, I tend to do old-fashioned work: I looked at the code that these websites serve up when queried.

They all (that I've seen) load big javascript libraries. Well ok then, that explains a good bit of what's going on.

Javascript - any browser scripting language - has some things it's allowed to ask the browser to tell it about the machine on which the browser is running. This helps the js (or other scripting language) understand what's available resource-wise, so it can optimise what it serves to the browser, from the server. Plus lots of other stuff it can do, some of which is really neat and some of which is fuel both for "browser fingerprinting" attacks and for horrific malware exploits (like last summer's #torsploit, brought to you by the NSA). So the javascript queries the browser about... DNS lookups? Well ok, but that's sort of orthogonal, isn't it?

Yes, it is.

In fact, when I poked at the js itself - and I'm far from a competent js coder, nor do I pretend to be one, thanks - it seemed clear that the ways this question is asked can only be described as hack-ish. Not that saying so is an insult; rather it suggests this is using tweaks to get an answer to a question that wasn't really expected in the design of the tool itself. Which makes sense: why would a webserver be entitled to know what DNS resolvers a client machine is going to use?

Further...

These DNS leak websites, as far as I can tell thus far, don't actually have the client machine do a DNS query and see where it queries .Nope, they ask the client machine - or some bits of the client machine - what DNS resolver it intendts to use, and then report that result as a leak. Or not. And this is, in a word, totally fucked when it comes to determining whether there is a "DNS leak" from an on-network client, or not. Think about it for a minute, visually. See?

Once you step back, it's obvious isn't it? The web server asks the browser to ask the client machine what DNS resolver it will use. The client machine - or some part of it - says "oh I use this/these resolvers for such questions" and hands it back to the webserver, via .js over the encrypted tunnel the answer. What the client machine doesn't say anything about is whether it would do this lookup within the tunnel or whether it would "leak" the query plaintext outside the tunnel... this is all a hypothetical answer to a hypothetical question. And there's nothing about the answer that says anything about whether it'd go across the tunnel, in any case. Which, by most relevant definitions, is what a "DNS leak" really means.

Even a little bit of wiresharking shows that the queries that local machines actually do aren't always matched up with what the "DNS leak" websites think the machines are going to do. Sometimes this variance shows a "leak" that isn't happening, and sometimes they show no leak when in fact the machine is really leaking plaintext DNS queries. These variances aren't 100% of the time, and the websites do a useful thing... but they are far from definitive, or really helpful in tracking down what the heck's going on.

Plus they have no idea whether queries go across-tunnel or not, since no actual queries get made during the "tests."

Right ok, then.

Now's where things get weird.

As I've been piecing together these bits of comprehension to a simple question that gets a little non-simple before getting outright pear-shaped (or perhaps cone-shaped?), anomalous results keep popping up that none of us can explain, even so. This goes on for weeks, as we come back to them, research, ask around to colleagues, come back, brainstorm...

Finally one of our member support staffers asks an insightful question: what about IP6?

Aha, there you go. We've written before, here about the issue of ip6 packet leaks in openvpn-based network frameworks. A known issue, and one we've been keen to tackle once the tools are there to do it properly (long story, separate post). A corresponding issue is, we now know, IP6 DNS resolver information leak... which is sort of the same thing, but also not at all.

Here's an example of what happens:

During cryptostorm session creation, the widget tells the operating system to shift over it's DNS resolution requests to the resolvers pushed down from the cryptostorm network. The operating system, in theory, does this and ensures that physical hardware (NICs) also has this new information, and uses it. This isn't always a perfect process, hence conventional "DNS leaks." But there's a secondary hole, too: what about IP6?

Because each NIC, each networking tidbit in fact, likely has its own preferences for what IP6 DNS resolvers to use. Perhaps those come from a local ISP, during DHCP, or perhaps from the local LAN router. Or perhaps from... who really knows? Those settings are rarely something folks put much thought into, as few folks are using IP6 intentionally yet, at the end-user side of things.

Well, then there's this: some browser (read: Chrome) apparently like IP6 alot so they go ahead and use their own IP6-based DNS resolver settings to look up even IP4 DNS questions for matters relating to web browsing resolution of namespace. So even if your OS, and your NIC, and your local router all know nothing of IP6 DNS resolution, it appears (from what the research says; we've not yet tested and worked with this in-house to best understand it) that your web browser might just decide to use IP6 to do DNS lookups. Which... wait a second. These DNS test website ask the web browser about what DNS servers the machine prefers, right? Right.

Yeah, now you see what's happening, a bit.

Are there "solutions" to this issue? Yes. There's tedious and imprecise ones that might break stuff. There's hypothetical perfect solutions that don't exist and might also have bad side effects. There's also a couple of elegant approaches we've put together, and now - before we test solutions - we're asking for help in validating what we think we're seeing here.

I'd hope to organise all this into a neat, screenshotted, how-to guide for folks to follow. That's not going to happen. Others on our team are great at that; I'm not. They are working other high-relevance tasks and can't pull off to make this pretty. So here's some links:

http://ipv6.google.com/
http://test-ipv6.com/
https://dnsleaktest.com
http://ipv6-test.com/
(and in particular: http://v6.ipv6-test.com/api/myip.php)
...this link randomly shows up in the browser window I've been using: http://wiki.wireshark.org/OpenVPN
(was there a reason, specifically? Anyhow it's useful info, so pasting it here(
and this is totally unrelated, but also relates to a possible solution, so why not post: http://code.kryo.se/iodine/

If you can make sense of what I'm trying to communicate in the post above, take a run at these test sites and see what happens. Turn off IP6 locally (this depends on your OS, in the details), Turn off IP6 on your NIC(s). Turn it off on your local router. Try mixes of those turn-offs and see which are relevant, and which oddly conflict. If you're feeling frisky, wireshark network sessions as you turn "6" on and off, at various levels.

Tell your browser not to use IP6 for resolution, especially if it's Chrome. See if that makes a difference in the "DNS leak" results.

See why this gets weird, and why we need more data before we can deploy a real solution? There's alot of variables; some, our in-house testing suggests, don't matter at all. A few seem to matter alot. We're not going to report provisional results, as we don't want to bias the data coming in from the community.

This isn't cause for outright panic. It appears that some substantial chunk of "DNS leaks" reported by some of the more clever websites are really artefacts of IP6 resolution that is theoretically possible on the client machine but is in practice rarely called upon in actual packet traffic coming off NICs. Some is, but not all of it. Are these "false positive" results, then? Not really, not the phrase I'd use. I'd say this is a complex question to answer and that the only real answer comes from sitting on the NIC and seeing what packets come and go: pcaps or STFU, in other words.

I suspect all this can be boiled down to a nice, concise explanation. I also suspect the solution to this "problem" is far more elegant than my rambling explanation is thus far. Which are both good things. But there's a long, looping path to get from there - a strange sense that these "DNS leak" websites are telling us something important... just not what they think they're telling us, nor what we first thought we're hearing (that's the "there"), to an elegant way to make sure no errant packets are doing stuff they shouldn't be doing in our security model (the "here"). Blah, that's an awful sentence - apologies.

And by all means, if there's nicely-written technical resources that lay all this out in beautiful form, share those links. I've hunted, and read, and sniffed around, and asked colleages to help me understand - as have several others on our staff - for quite a few weeks. Lots of general papers on IP6 or DNS resolution or whatnot (of course); not so much on "this his how browsers answer questions about DNS queries on client machines, and how those answers might or might not relate to actual packets doing actual things." We'd love to see more of that, please! :-)

There's quite a bit of stuff out there written about "DNS leaks" - by VPN companies, by review sites, by random folks. I'm discomfited by the fact that I've not seen these browser-side questions tackled by someone smarter than me, already, and posted publicly. It makes me sense I'm off-track... but we've poked at these wireshark results enough and I've done enough testing with those site links, above, to know something is happening beyond the simplistic explanations posted out there thus far. And I do feel that there's been some (inadvertent or otherwise) tuning of VPN client software to get "green check" passing results from several of the DNS leak websites, with all but total ignorance of why those green checks happen, or not... or whether they relate at all to actual packet traffic. That's a sobering realisation.

Bad news? Right now, your machine is likely leaking some IP6 packets you don't know about, whether you're on cryptostorm or not (folks blocking IP6 at their router, or similarly hands-on tactics, likely are not... but not guaranteed, actually; more on that later). This has been known and flagged by us since our launch, in the general category of "IP6 leaks." Good news is that we can button up those leaks, not just the generic class but also the (I suspect far more pernicious and subtle) IP6 DNS query flavour themselves.

tl;dr throw some data at us, and we're going to present to the broader community some tools to improve real-life network security that nobody's ever created previously.

So... that's that.

Cheers,

    ~ pj

User avatar

parityboy
Site Admin
Posts: 1105
Joined: Wed Feb 05, 2014 3:47 am

Re: DNS weirdness study guide & mega-thread aka Cthulu

Postby parityboy » Fri Nov 28, 2014 3:45 pm

@PJ

This is where the interesting questions start.


Fixed. :P

Something I've just noticed; the "nodelist.txt" file used by the Windows widget uses IP addresses rather than FQDNs, so assuming that the DNS resolvers pushed down the tunnel are correctly applied to the resolver mechanism on the OS, the issue of pre-tunnel out-of-tunnel leaks actually has a much greater chance of affecting "native" OpenVPN users (specifically Linux and OS X) - those are the platforms which use the OpenVPN config files.

I've known for a while that the /etc/resolv.conf file on my system doesn't get updated by Network Manager even though the OpenVPN entry has DNS resolvers specified. It's less of an issue because I don't use HAF entries for exit node resolution, and the resolvers I do use are ones preferred by CS anyway.

With regard to the DNS leak tests, what I see is missing from test mechanism (going by your explanation) is

1) ask the client browser for its default route (can JS do that?)
2) make a connection to the DNS resolver (presumably along that default route, but not 100% guaranteed) and resolve a known address such as "www.google.com". (possibly also do a traceroute?)
3) report back results to the web server.

If a tunnel is active, any DNS resolver presented by the client browser should be accessible along the default route AND should not have an address of 192.168.x.x (that would make it a local out-of-tunnel resolver, i.e. possible leak). Obviously, it's far more difficult to account for things like split routing, but I believe it would paint a far more accurate picture of what the client is actually set up to do, rather than simply asking it. :)

User avatar

Pattern_Juggled
Posts: 1492
Joined: Sun Dec 16, 2012 6:34 am
Contact:

deeper-dive network path analytics

Postby Pattern_Juggled » Sat Nov 29, 2014 1:38 am

parityboy -

I think we're going to get some rather useful data from these two tools; figured I'd share, so you can take a poke at 'em, as well:

http://netalyzr.icsi.berkeley.edu/index.html
http://mlab-live.appspot.com/tools/ndt

Note that both are Java-based "browser" tools that are much more along the lines of full-bore network analytic packages thinly wrapped in a browser skin; for folks who might want to work with them, understand that there's much more involved, security-wise, in providing sufficient permissions to run these tools than is (in theory) required for more commomplace, run-of-the-mill javascript-based applets.

Cheers,

    ~ pj


Return to “DeepDNS.net - cryptostorm's no-compromise DNS resolver framework”

Who is online

Users browsing this forum: No registered users and 2 guests

Login