Ξ welcome to cryptostorm's member forums ~ you don't have to be a cryptostorm member to post here Ξ
Ξ any OpenVPN configs found on the forum are likely outdated. For the latest, visit here or GitHub Ξ
Ξ If you're looking for tutorials/guides, check out the new https://cryptostorm.is/#section6 Ξ

forum upgrade to 3.1.1

Looking for a bit more than customer support, and want to learn more about what cryptostorm is , what we've been announcing lately, and how the cryptostorm network makes the magic? This is a great place to start, so make yourself at home!
User avatar

Topic Author
cryptostorm_admin
ForumHelper
Posts: 74
Joined: Tue Jan 01, 2013 5:43 pm
Contact:

forum upgrade to 3.1.1

Postby cryptostorm_admin » Mon Nov 03, 2014 11:35 am

This Saturday evening, the admin team decided to go ahead and upgrade the backend software of the forum (phpBB) to the newly-released version 3.1.1. This is always a balance, and there's pros and cons to any fairly major upgrade of this sort.

Pros:
    1. Often improved security from bugfixes and code cleanup in newer versions (true in this case)
    2. Better performance and/or scalability in new versions (somewhat true here)
    3. New features available (not true here; see below)

Cons:
    1. General chaos during the upgrade process (pretty much always true, despite efforts to minimise)
    2. Risk of data loss if things go horribly wrong (easy to mitigate with realtime backups; see below)
    3. Disharmony with earlier tweaks or edits to source code (yep, see below)

We scheduled the update for Saturday evening EST, as that's often a (fairly) slow time around the forum. What follows is a tl'dr summary of things from there.

Full backups of the earlier install have been made, and we apply the 3.1.1 upgrade. Things are fairly smooth, although in the new version there's a paucity of custom styles available yet, as it's quite new into full production. So we can't bring forward the old (actually fairly new) forum board style, and we install a new style which, on balance, we think is rather nice.

However, the new version of the forum is totally non-compatible with all the old extensions and modifications to 3.0.x versions. In fact, having any traces of old mods/extensions anywhere in the pre-upgrade production environment is a Recipe for Disaster. Which is not terribly well-documented in the upgrade literature, to be blunt. But that's how upgrades go, sometimes.

Thus, since early Sunday morning we've been left with one tedious bug involving newly-posted materials (or edited prior posts) that contain bbcode markup (italics, boldface, URL links, etc.): an ugly little cascade of php errors spits out during previews or posts, and the newly-posted material will not display bbcode markup properly (although unedited prior posts display fine).

Extensive research since then confirms that this is a known "issue" (as the wording goes), and that many other folks also experience it. Our admin team has been experimenting with various fixes since then, after having confirmed a couple of important data points:

    1. the newly-made or newly-edited posts are saving fine to the database (mysql) and there's no referential integrity or data loss problems at play; this appears to be purely a display-layer artefact of the upgrade

    2. there's no apparent security or instability risks being brought to the production version of the forum at this point in time; this is a mildly annoying, but not serious, bug we're chasing.

    3. we feel there's a fairly simple fix for this involving removal of some "orphaned" mod entries in one of the database tables; we're not rushing frantically to force that fix into production, had have taken instead some time to research the situation and ensure there's no high-impact issues we need to be concerned with first.

So, in summary: we have full backups of the entire prior (3.0.x) forum install, and in a pinch can simply revert to that (with all new posts retained in the database tables), if we hit a proper wall in debugging. But we're not feeling like that's likely, and instead are working closer to cleaning out this issue in the upgraded forum backend.

As the forum is not network-core (technically speaking) we've not pushed aside other work to quash this bug earlier today. Our apologies to forum moderators and members who have been frustrated by it today - it looks worse than it really is, and in fact all posts and the bbcode markup in them are saving fine and will display fine once we track down this zombie mod DB entry. So there's no work or content lost... it's just an inelegant day of poorly-displayed (new) posts.

Our thanks for the patience folks have shown. We've been hesitant to do this upgrade to the forum, but on balance decided the potential (and real) hassle is worth the improvements and bugfixes in the 3.1.1 framework overall (it's got some nice consolidations of very brittle components of the older architecture, which over time will make new mods and new styles far more robust and far less likely to cause interoperability problems, which has been a serious issue with phpbb for many years). We still think that's the case, even after a day of on-and-off fiddling with this one annoying bug.

There is an open forum thread to discuss and expand on this informational post, so this one will be locked in order to keep the discussion rolling in a more comfortable local. Also we'll update this post once the bug is squashed (and likely merge it back into the aforementioned thread).

Thank you,

    ~ cryptostorm_admin

User avatar

df
Site Admin
Posts: 371
Joined: Thu Jan 01, 1970 5:00 am

done

Postby df » Mon Nov 03, 2014 12:24 pm

Bug is squashed. Forum upgrade complete. On a side note, and just to make everyone happy, I'd like to mention that we took our time upgrading to the latest version mostly because exploitation of all of the known vulnerabilities that were present in the other phpBB version we were using would have have been stopped by our mod_security (https://www.modsecurity.org/) rules and suhosin (http://www.suhosin.org/stories/index.html) settings. Plus, we do execv() syscall logging, so if any of our httpd (apache) processes started executing unusual external commands all of the sudden (as is common with RCE-style attacks), we would notice. Also, since we use suexec (http://en.wikipedia.org/wiki/SuEXEC), on the off chance someone was able to execute commands, they would only be able to affect the files of the vulnerable website and not any other website on the system. So hacking another website on this system won't get you into the forum.

We haven't seen any of the aforementioned activity in any of the logs (excluding some stuff mod_security stopped that was most likely automated worms), so you're still safe :D


Return to “cryptostorm in-depth: announcements, how it works, what it is”

Who is online

Users browsing this forum: No registered users and 10 guests

Login