The forums at support.bryght.com were getting hit pretty hard with spam over the last couple of days, so I took the executive decision to temporarily close them. I'm committed to working on documenting what's happening so that we can feed back to the Mollom anti-spam project (see also mollom.com) so that automated posters are dealt with. If you have support issues that you would normally post to the forum, until we can resurrect them, please use the contact form and we'll look at your issue via our email ticketing system.

Last night Roland ran the database maintenance on newer hosted service sites powered by Drupal 5. We are looking at doing another pass soon, because today we noticed there are a few other things about the newer databases that we can tune up.

While some sites are singing along happily today, we did notice that sites that use the Image module heavily, especially those that use the Random Image block, are suffering performance issues. Today we upgraded the version of the image module to its latest official release, which seems to have fixed an issue with thumbnailos, but I'm not convinced that it improved performance much. Specifically, sites that use the Image module in Drupal 5 are experiencing Internal Server 500 errors, but for reasons different, I suspect, than database performance issues that plagued us in the recent past. We are continuing to investigate this specific issue.

After running the database maintenance, details of which are below, on over 350 sites over the weekend, we noticed an increase in speed for our flagship site bryght.com (its database was reduced to 1/10 its original size!) as well as some of the sites we targeted for special attention. We're still waiting for word from those we've contacted specifically, so until we hear from them, we're not ready to say what we've done has successfully rooted out the 500 error problems people have been seeing on the hosted service. (We're waiting to hear from them as they are the people who watch over the site the most.) This evening we'll run the database maintenance on our newer sites, but we don't expect to see as much improvement as we did with the older sites, partly because the newer sites handle their individual databases better, and because at the outset we had the "Database Logging" module disabled by default. The equivalent module on older sites was filling up databases, making them run slowly for many operations.

A note on what we did to older databases, which we'll also do for newer sites tonight: we cleaned out the following database tables, which don't contain any content but rather user login sessions and old system logs: accesslog, watchdog, and sessions. Clearing out the last one means everybody on the site will be logged out. We hope that the inconvenience of having to login again is outweighed by a faster-running site overall. About 180 sites will undergo maintenance tonight, but like I say, we don't expect the performance gain to be the same as with older sites.

At any time during the day, please let us know if you encounter Internal Server 500 errors and what you were doing when it happened so we can take a deeper look. We will update this post with a comment when we're done with tonight's maintenance, and follow up with a post tomorrow morning.

We have the database maintenance scripts written, and this morning we tested them out on one of our older sites (that we built for ourselves). We did say that we would do the maintenance overnight, but we were so happy with how fast and smoothly the test went that we decided to run it on 100 active sites on this, a Friday afternoon, and evaluate from there. We make backups on every site before proceeding with the maintenance, so no content will be lost.

If you see something unusual or unexpected on our site, please continue to report it and we'll look at it right away.

A firewall connectivity issue caused by an errant system overloading our network pipe resulted in a two hour outage on all Bryght Light sites and Bryght VPSes. We have blocked the errant system. Please let us know if you experience any further issues.

Yesterday morning everybody on the hosted service experienced their site going down, and since then sites have experienced numerous 500 errors. (Some sites have experienced this off and on for the past few weeks as well.) We are currently finishing with the checks of all databases on the hosted service, and then we will run some proactive maintenance—i.e. deleting the data from sessions, accesslog and watchdog tables that have not been automatically cleared like they should have. Don't let that stop you from reporting errors you see that you think are out of place. If you see "500 error", please send in a report with the URL of your site and we can double-check that you're on the list for the planned maintenance.

All Bryght Light hosted sites should be back up.

All (or most) of the sites on our hosted service are currently showing 500 Internal Service error. We've isolated the problem, and the SQL server is working its way back from an aborted maintenance. We'll have an update here when it's back and running.

Update about a half hour later: all sites are back up.

As per the Drupal 5.3 announcement, we've released as tag 2007-10-24s. The update contains security patches and minor bug fixes to the Drupal core. There are no changes to or additions of modules since the previous tag.

This release does not have any database updates, but it's always a good idea to backup your site and then run the update.php script, just in case you missed some the last time around. It's all in our instructions in maintaining your version of Drupal 5 on your VPS.

Note that if you see a Google Analytics error at Administer (top menu) » logs » Status report, i.e. "Google Analytics module has not been configured yet. Please configure its settings from the Google Analytics settings page.", you can fix this by turning off the Google Analytics module (if you don't use it) or by clicking Administer (top menu) » Site configuration » Google Analytics. See our documentation on the Google Analytics module for help in configuring it for your site.

So, we're back. As you will have noticed, we were down for multiple hours today. The data center where our servers are located, and managed by the fine folks at WorkHabit, had their power supplies essentially melt down. A cascade happened which somehow caused the triple redundant power systems to overload and shut down one after the other, happening so quickly that the emergency generators didn't even have time to kick in. In short: the entire data center went dark in a few minutes.

This event also happened to take out a chunk of phone service -- including 911 for several hours! -- in the area (that's just us trying to feel better that the "big guys" took a hit as well). In any case, a full scramble went on at the data center, the system was plugged back into the city's electric grid, and systems came back up as we had people onsite manually restarting each machine. We expect that there will be some cleanup to do around database corruption issues: if you notice anything strange, please don't hesitate to email support, we'll be doing database checks over the next couple of days.

Also, now that our friends at Jaiku were bought by Google, looks like lots more people are checking them out. We posted a couple of updates as they happened to the Bryght channel on Jaiku -- we'll map that to some Bryght URL so it's easier to get to, but in the meantime you early adopters out there might want to check it out.