Ξ welcome to cryptostorm's member forums ~ you don't have to be a cryptostorm member to post here Ξ
∞ take a peek at our legendary cryptostorm_is twitter feed if you're into that kind of thing ∞
Ξ we're rolling out voodoo network security across cryptostorm - big things happening, indeed! Ξ
Ξ any OpenVPN configs found on the forum are likely outdated. For the latest, visit GitHub Ξ

Redundancy in website, email, & IRC infrastructure (etc.)

Looking for a bit more than customer support, and want to learn more about what cryptostorm is , what we've been announcing lately, and how the cryptostorm network makes the magic? This is a great place to start, so make yourself at home!
User avatar

Topic Author
df
Site Admin
Posts: 283
Joined: Thu Jan 01, 1970 5:00 am

Redundancy in website, email, & IRC infrastructure (etc.)

Postby df » Thu Jun 04, 2015 3:20 am

[i]{merged several related threads into one, for ease of access ~admin}[/i]


Our server in Iceland that hosts this forum and cryptostorm.is went down last night, as did the server hosting the Iceland exit node.

The remote logs showed no sign of any intrusions or attempts, and after a lengthy email conversation with a few of the good people at Datacell (who host our Iceland servers), it turns out there were routing issues that arose when a new link was added to the network.

During the downtime, we temporarily loaded our cryptostorm.is backup onto the same server running cryptostorm.nu, which is why cryptostorm.is started resolving to 212.83.185.245 (and was giving an SSL cert warning because the SSL cert in use was for cryptostorm.nu not cryptostorm.is).

We were also in the process of setting up some more redundant backups using VMs on a dedicated server in Moldova, but those VMs weren't completely provisioned in time for this Iceland downtime.

Anyways, Datacell has fixed the routing issues, so this server (and the Iceland VPN node) is back up and the DNS for cryptostorm.is has been switched back to 79.134.255.38.

User avatar

Operandi
Posts: 88
Joined: Fri Nov 22, 2013 4:23 pm

Re: Iceland went down last night, back up now

Postby Operandi » Thu Jun 11, 2015 11:42 pm

The Iceland node has been dead for me for a couple of days now. I can connect to it, but no sites will load (except for https://cryptostorm.is/test).

Frankfurt and Montreal have been somewhat unstable lately, as well.

User avatar

Topic Author
df
Site Admin
Posts: 283
Joined: Thu Jan 01, 1970 5:00 am

Re: Iceland went down last night, back up now

Postby df » Thu Jun 11, 2015 11:55 pm

We haven't noticed any problems with Montreal. Frankfurt's ISP hasn't mentioned any downtime in the links that control our VPN node, and they're pretty good at notifying all customers about every little hiccup going through the server (even for links that have nothing to do with our server).

As for Iceland, I can confirm from several different networks that the website is now up.
If you can access https://cryptostorm.is/test then you can definitely access the other parts of the website.
Maybe your browser is reading from cache the error it saw when Iceland was down?
A good way to verify is to use a command-line thing like wget or curl to verify you can access cryptostorm.is.

Even if your router/OS is caching the old DNS record of cryptostorm.is, that should work too since I left the backup server running on the nginx handling cryptostorm.org, so if Iceland goes down again all I have to do is change one A record to have the traffic go to the backup system.


DudeOfLondon
Posts: 109
Joined: Sat Jan 10, 2015 5:14 pm

Re: Iceland went down last night, back up now

Postby DudeOfLondon » Sat Jun 13, 2015 1:25 am

Frankfurt node is very picky since yesterday. Every 15-20 minutes I have massive timeouts for about 2 minutes and then it mostly gets back to normal.
When the timeouts while browsing appear, I made ping tests.
Pinging the LAN-router is 1ms.
Pinging my own External IP: 46.165.222.245 gives ~55ms
But pinging goolge.de for example gives timeout.

User avatar

ntldr
ForumHelper
Posts: 39
Joined: Sun Feb 01, 2015 4:15 pm

Re: Iceland went down last night, back up now

Postby ntldr » Sat Jun 13, 2015 11:14 am

DudeOfLondon wrote:Frankfurt node is very picky since yesterday. Every 15-20 minutes I have massive timeouts for about 2 minutes and then it mostly gets back to normal.
When the timeouts while browsing appear, I made ping tests.
Pinging the LAN-router is 1ms.
Pinging my own External IP: 46.165.222.245 gives ~55ms
But pinging goolge.de for example gives timeout.


https://twitter.com/cryptostorm_is/status/609373924137304064

here's the problem :/

User avatar

Operandi
Posts: 88
Joined: Fri Nov 22, 2013 4:23 pm

Re: Iceland went down last night, back up now

Postby Operandi » Fri Jun 19, 2015 2:31 am

df wrote:Maybe your browser is reading from cache the error it saw when Iceland was down?

Unlikely. As far as I recall, I could use the Iceland node just fine after it went online on the 3rd of June (and before it suddenly stopped working for me).

But anyway, I tried to connect to it the other day, and it seems to be working properly now. The connection feels a bit slow, though.

User avatar

jlg
Posts: 92
Joined: Mon May 05, 2014 2:44 am

Re: Iceland went down last night, back up now

Postby jlg » Sat Jun 20, 2015 4:07 pm

I personally have been sticking away from Iceland node for a while now. Too many issues for my liking of internets.


gbj
Posts: 20
Joined: Thu Mar 27, 2014 8:22 am

Re: Iceland went down last night, back up now

Postby gbj » Mon Jun 29, 2015 4:43 pm

Fernrir has been down for the last five days for me and I get no reply when contacting cryptostorm support :(

User avatar

cryptostorm_team
ForumHelper
Posts: 159
Joined: Sat Mar 02, 2013 12:12 am

Redundancy in website, email, & IRC infrastructure

Postby cryptostorm_team » Mon Jun 29, 2015 10:06 pm

During the past several weeks, we've accelerated a longer-running project to add redundancy and resilience to our websites and other non-network resources (we call these "non-network" because these items are not part of deliving cryptostorm's secure network itself, which is entirely separate from any websites or other single-point-of-failure components).

In the first two years of our existence, we didn't judge the need for such capacity to be mission critical; a small bit of downtime here and there with cryptostorm.is, for example, might be a minor inconvenience for all of us but would not be critical path. Of course, we retain rolling backups of files (and most of our website source is already hosted at github and thus is on independent infrastructure), so in the event of a sustained outage we could - and several times, did - switch over to secondary server capacity with the backup images.

Most companies handle this issue by outsourcing their hosting to a "content delivery network" like Cloudflare. For a basket of reasons too long to list here, this is not an approach with which we are comfortable, though it is "easier" and for less technically centred project teams it will in many cases be too tempting to pass up.

So, as cryptostorm has grown and evolved since 2013, we've known that the need for redundant website (and email, and IRC... we'll just say "website" and assume all that is included, as well) capacity would eventually be something we'd need to address. As we discuss in a bit more detail in a parallel blog post at cryptohaven.net, recent attacks on Iceland's internet infrastructure have caused access to our websites (which have always been hosted there, with our colleagues at Datacell) to become, in a word, sporadic (through no fault of Datacell's, to be clear).

Given that, we pushed forward to complete our internal effort to provide redundant, distributed, failsafe website access - we'd been making steady progress but with no deadline in sight, it naturally slipped behind critical tasks and was in some senses sleepwalking. Issues in Iceland got things into fast gear, and we set a tight timeline to get things in place.

Two days ago, on Saturday, we did our first production cut-over test of the new model we've put in place. Most went smoothly, and our security procedures held together comfortable. However, there were the (if we're being candid) expected hiccups here and there: the database powering this forum was intermittently refusing to stay up on Sunday evening, for example. Those issues are all now resolved and we're fine-tuning the details.

In this thread, we'll post a bit more technical detail on how we've approached this infrastructure redundancy project - some of it's a bit routine and boring, but other components are perhaps novel and even somewhat elegant in final form. It's worth nothing that the overall project is not complete; what we've done is the first cut-over test. Now, we're layering in the automated redundancy itself (in technical terms, the first step was actually more of a challenge than the redundancy itself).

Finally, it appears that our automated 'tokenbot' delivery of newly-purchased tokens was inactive from early Sunday through Monday morning. We'd concluded this was merely the result of cached DNS data in email delivery systems, but that conclusions was not accurate and in fact the tokenbot was simply not delivering tokens. Since then, we've manually confirmed all tokens not delivered timely during that period have now been delivered. Further, we've provided complimentary 66-day tokens to all those members affected by the delay. This was a genuine screw-up on our part - timely token delivery is a big deal to us, and to many members - and we offer our apologies for not being aware of the issue, and resolving it, sooner.

If there's additional questions or reports of transitional bugs, please do feel free to post them here - we'll do our best to stay current with replies. Through today, we've invested substantially all available team effort in completing the first step of this project, and thus haven't posted much data here on what's been in process. Now that's complete, we're able to do a better job of keeping the membership informed as to ongoing developments.

Best regards,

~ cryptostorm_team
cryptostorm_team - a shared, team-wide forum account (not a person)
PLEASE DON'T SEND PRIVATE MESSAGES to this account, as we can't guarantee quick replies!
--> feel free to use any of our other contact channels, or post in the support forum
cryptostorm: structurally anonymous, token-based, unlimited ☂ bandwidth, opensource, darknet data security for everyone!
keybase.io validatorsonename.io validatorsPGP key @ MITnetwork statuscryptostorm github
support team bitmessage address: BM-2cTMH8K5JnjbfSALjZtSkRWCLfc3Tr8GBV
support team email: support@cryptostorm.is
live chat support: #cryptostorm


gbj
Posts: 20
Joined: Thu Mar 27, 2014 8:22 am

Icelandic server

Postby gbj » Wed Jul 01, 2015 9:04 pm

Has the Icelandic server been taken down. I dont even see the fenrir exit node on pingdom.com. And this is after a week of downtime :problem:

User avatar

Pattern_Juggled
Posts: 1492
Joined: Sun Dec 16, 2012 6:34 am
Contact:

re: Iceland & pure.cryptohaven.net

Postby Pattern_Juggled » Mon Jul 06, 2015 11:45 am

We have been integrating a new, less technically intense platform over at [nb]pure.cryptohaven.net[/b], and to be honest we're still learning how to coordinate information posted there with threads here.

In this case, we provided an update on Fenrir and associated Icelandic infrastructure at cryptohaven last week... but failed to provide an echo or reference to that information here. Seems obvious, in hindsight. It's not clear if we'll be automating that coordination, or simply manually echoing - either way, it's something that will be done.

Meanwhile, here's an old-fashioned copy paste of the relevant data from cryprohaven's post on the subject:

...

This process has been underway for more than a month, as we saw the need to provide redundant capability to serve our websites... and it was more or less on track when, in recent weeks, problems with the internet connectivity coming and going from the island of Iceland started to become noticeable, then common, then almost overwhelming... in the past couple days, "overwhelming" is the description best suited. A visit to our cryptostorm.science network status page shows all the gory details, which impact both our websites and our Icelandic cluster (anchor node: fenrir).

Because these problems are 'upstream' from both our servers themselves, and from the datacenter in which our servers are housed, there's little or nothing any of us can do to resolve them. It's like having construction on a highway between one's house and one's intended destination: no amount of driveway sweeping or cleaning will help with the highway's crash site, and until that bottleneck clears there's not going to be much happy motoring to be had.

We're not leaving Iceland, and we're not leaving our current datacentre there! However, reality is that availability there is taking a hit lately - being an island, that's a risk. Word 'on the street' (i.e. amoungst well-connected colleagues in the deeper parts of the security tech ecosystem) is that these attacks relate to certain governments trying to "break" the anonymity of visitors to some sites within the Tor privacy network. We'll write a bit more about that in a separate post, but if that's why Iceland is being hit so hard lately, it's doubly tragic: both for the targets of the attack on Tor anonymity, and because the entire country of Iceland is being impacted so one vendetta can be acted out.

We'll update this post once the sites roll back to their normal selves... meanwhile, feel free to read the couple of posts here at cryptohaven. Not much, yet, but we're happy so far with how the project is progressing.


There's not much more to add, in terms of Iceland, meanwhile - we've prioritised the infrastructure redundancy effort as critical path since then, and largely focussed on ensuring it was completed with minimal drama. That process, although still a couple steps away from its final state, largely in-hand (knock wood) and having completed the gnarly chunks of it, the admin team has been provisioning several new nodes and an entirely new mechanism for secure session routing, this holiday weekend.

Once that's in hand and rolled out early this week, we'll be circling back to see what we can do to re-launch our icelandic cluster in a way that maximises performance, resilience, and security. It's too early to say with certainty, but we're cautiously optimistic that we can do this without either spending gratuitously for very little member-supporting capacity and without sacrificing session security in the process.

Regards,

~ pj
...just a scatterbrained network topologist & crypto systems architect……… ҉҉҉

    ✨ ✨ ✨
pj@ðëëþ.bekeybase pgpmit pgpðørkßöt-on-consolegit 'er github
bitmessage:
BM-NBBqTcefbdgjCyQpAKFGKw9udBZzDr7f

User avatar

sin
Posts: 8
Joined: Mon Jun 01, 2015 4:55 am

Re: Icelandic server

Postby sin » Tue Jul 07, 2015 11:59 am

I vote to booby trap iceland datacentre with thousands and thousands of tacs placed strategically around the compound.


Return to “cryptostorm in-depth: announcements, how it works, what it is”

Who is online

Users browsing this forum: No registered users and 17 guests

cron

Login