Undrblog

I run a number of VM’s under KVM.  Windows VM’s, RedHat VM’s, SLES and Ubuntu VM’s.  Recently, we attempted to provision a VM using LANDesk.  We ran into an issue where KVM was not assigning a BIOS serial number.  LANDesk uses the BIOS serial number to identify the client to the LANDesk server.   I worked…

Please bear with me. After a SECOND almost successful attempt at compromising my server using flaws in Serendipity, I’ve decided to move to Drupal. You’ll find the old blog entries here, but not much else. I’ll get more moved over as I can.

I found myself at work the other day, without my keys. Luckily our offices share the same key, so a co-worker was able to get me into my office. Not so my desk though. I don’t keep anything terribly sensitive in my desk, because i’ve always felt the lock was a tad on the cheap…

“Here we are again, on this rust-bucket of a boat.” thought James, as he looked around at the poorly supplied vessel he found himself on. Hardly any ammunition, a single torpedo, practically defenceless. “It’s a good thing I’ve got Wren to look out for me…”. James, and two other mercenaries were contracted by a ships…

Over the past 20 year or so of my life, I’ve been tinkering with these things called computers. I started out when i was young, in my parent’s attic, on my Dad’s Atari computer. He had a simple system, one 5.25″ floppy drive, and a hand full of software for it. He used it for word processing, and a hand full of games. I can vaguely remember the first time we sat down, and he showed me how to load up a game off of one of these floppies. The procedure involved in turning the thing on, and when the disk had to be put in, when to push the power button all of that. I cant remember how old i was, but it had to be before i was 9. I could have been… 6, maybe 7? I’m not trying to get nostalgic, this is back story, It’ll make sense in a minute.

…

For months I’ve been following the Diaspora* project. On 9/15/2010, they opened up their git repository, so that eager IT people like me could go grab their code, and start poking at it.

So, on 9/15, I watched their site eagerly waiting for the announcement. Sometime late evening (around 8-9PM EST) it finally popped up, the link to the git repo. I donwloaded it within minutes, and started tinkering on my laptop (Fedora 13), and had it up and running in about an hour.

I thought I’d write a little blog entry about the setup, and my initial thoughts.

First, i’ll say that Diaspora* runs on top of some rather new software. Fedora is a testing bed, it’s usually full of bleeding edge software, things that are being tested and vetted for eventual use in RHEL. I’ve had issues in the past where the versions of software in Fedora were so new, they were alpha, and i actually had to downgrade them in order to get things working.

Not so with Diaspora*. Diaspora* is written in ruby, and its data is stored in a noSQL implementation called MongoDB. There are no packages for MongoDB for Fedora, and the version of Ruby packaged in Fedora is one sub-revision too old. None of this was a huge issue on Fedora however. I grabbed Fedora Rawhide rpm’s for Ruby 1.8.7, and MongoDB is distributed in binary form, so i just grabbed the binaries and put them in place, which is actually detailed in Diaspora’s README.

This was all well and good, but i wanted to run this on a web server, not my laptop. I like Fedora for my workstations, but for a server.. I like things a little more conservative. So i run CentOS for all of my personal systems, and RHEL on all of the systems at work. Both distributions are similar, in fact, CentOS is based on RHEL, its just free, where RHEL requires a subscription. Generally, there are compatibility issues with bleeding edge software like Diaspora, and RHEL and its clones. This is because when a package reaches RHEL, its supposed to be completely stable. Bleeding edge is rarely completely stable.

So I built myself a new VM, which will host Diaspora. At this stage, I don’t fully trust the security of Diaspora, even the developers have come out and said that they know there are security issues with this release, but its pre-alpha! The VM makes it segregated, and means I can play with packages that I may not be willing to do on my production web server. If i hose it, just junk it and start over.

What I started with

CentOS 5.5
1GB of memory
1CPU
40GB Disk
Hypervisor: xen

Extra repositories

EPEL
RPMFusion (Free and non-free)

CentOS Install

Nothing special here, i installed the bare minimum, just the base packages, and networking.

Preparation

Once the OS was installed, I ran a yum upgrade, to get the latest packages for the base install, and rebooted.

After the reboot, i installed the following from yum:
screen
openssl
openssl-devel
git
vim-enhanced
gcc
libxslt
libxslt-devel
libxml2
libxml2-devel
ImageMagick
gcc
gcc-c++

You’ll notice that Ruby isnt in that list. Thats because ruby is at version 1.8.5 on CentOS, we need 1.8.7.

So, i installed the following from Source:
ruby-1.8.7-p302
rubygems-1.3.7

I ran into an issue when I built ruby that ruby-openssl didn’t build. I think this was because I didnt intially have openssl-devel installed. I added it later, and then built ruby-openssl by running “ruby extconf.rb” in (ruby source dir)/ext/openssl. This built and I was on my way. If it doesn’t build for you, you’ll find out when you try to launch Diaspora for the first time. if it gives you an error about openssl being missing, make sure openssl-devel is installed, and then build it for ruby as I’ve mentioned.

Now, we need Ruby’s bundler gem, so install from gem:
bundler (gem intall bundler)

Setting up MongoDB

This is noted in the Diaspora* README.md.

Essentially, unpack the MongoDB binary package, and then copy everything in (mongo directory)/bin to /bin on your system.

Then create /data/db with “mkdir -p /data/db”.

The Diaspora directions say to chmod -R 777 /data, but this seems insecure to me. mongod runs as root, so I gave some tighter perms a try. /data is owned by root:root, i set permissions to 700 recursively, mongo appears to start, and Diaspora functions.

Starting mongo is as easy as running “mongod” as root. I send it to the background so that I can free up my terminal.

Setting up Disapora* itself

If you don’t already have it, go get Disapora’s source. You can get it at GitHub or via git on the command line with “git clone http://github.com/diaspora/diaspora.git”. I put Diaspora in /var/diaspora.

Once you have the package, run “bundler install” in diaspora’s root directory.

If you run into errors here, you’ll need to track them down. Most commonly its caused by a missing package, or some other mis-configuration.

Once that finishes, you;re just about ready to run diaspora!

Running diaspora

Diaspora runs via ruby’s “thin” http server. To start diaspora run “bundle exec thin start”. You’ll see something like:

>> Using rack adapter
>> Thin web server (v1.2.7 codename No Hup)
>> Maximum connections set to 1024
>> Listening on 0.0.0.0:3000, CTRL+C to stop

This means that Diaspora is up and running! You can connect to it via http://yourserversip:3000

Some notes on running diaspora.
To run on standard port 80, you’ll need to add a -p 80 to your command line. “bundle exec thin start -p 80”
Also, i like to run it via screen, so i can detach it, and re-attach it later. “screen bundle exec thin start -p 80”
Then once its started, use ctrl-a, ctrl-d to detach from screen, then screen -list to find it again, and re-attach with screen -r (id from list).

Using Diaspora*

Now you should be to a point where you can actually login, and use Disapora.

Pointing a browser to your install will get you to the login page, first you’ll need to sign up. There’s a link for that on the page. Once you’ve signed up, if you intend for this to be your own little playground, and not open to registration, you can disable new signups by following the entry in the Diaspora FAQ regarding disabling outside registrations.

Keeping Diaspora up to date

If you’re running this pre-alpha code publicly, I’d highly recommend keeping it fresh with what’s available on the git repo. You can download it manually, or if you downloaded it via git in the first place, you can use “git pull” to grab the latest via the command line. Once you’ve done so, you’ll need to stop the server, and run “bundle install” again, and then “rake”. You could run this via a scheduler. I intend to write a script which does all of this. I’ll post it whenever i’ve done so.

A quick and dirty updater script, as promised.

[root@pod ~]# cat bin/updatediaspora.sh 
#!/bin/bash
cd /var/diaspora
/usr/bin/git pull
/usr/local/bin/bundle install
/usr/local/bin/rake
/bin/kill `ps awx | /bin/grep 'thin start -p 80' | /bin/grep -v grep | /bin/grep -v SCREEN | $AWK '{print $1}'`
/usr/bin/screen -d -m /usr/local/bin/bundle exec thin start -p 80

Conclusion

I really like this project thus far. I’d like to learn more about how things are going to be linked later. Right now i’m unable to link to users on any other systems, I don’t know yet if this is an issue with the state of the project (i.e. not yet implemented) or a config issue on my part.

I have a few co-workers and friends using my install, its fun to toy with, I’ve not yet dug into any of the code. i don’t know ruby, so I’ll need to learn something about ruby before I’ll be able to make much sense of the source.

I like the idea of aspects for separating co-workers from friends, from whatever else. It does seem like this could get confusing until you get used to the idea however. The UI is nice, i like the styling and it doesn’t feel overly clunky. Which is saying something this early on. All in all, I can’t wait for this project to pick up more followers. …

I’ve had my blackberry storm for a little over a year now. I’ve decided to follow up on my previous posts regarding this phone and my opinion of it..

What this phone is good at:

Email, that’s about it.
Its interaction with BIS to handle email is pretty trouble free. Except if you’re setting up multiple accounts, I actually had to call Verizon and have them reset my login because the setup wizard creates a basically useless accound on the BIS, to add a second you here to get that unlocked. Which brings me to the relative stupidity of this email model. Yes, it works, and works well once configured, but why do things this way? Why can my phone not talk directly to my e-mail server? What is RIM protecting it from? I’m an e-mail administrator, in the case of every e-mail address I use, I am in complete control of both the server, and the client. Why can’t this be done on my phone as well? Instead RIM needs to put their server in the middle? Still, this aside, the blackberry does do email well.

What this phone is not good at:
Everything else.
Seriously, I’ve not found one other thing that this phone does well enough that I’m willing to look past its flaws.

Web Browsing
The browser is slow, clunky, and is actually painful to use. I rarely use the browser because of its difficulties.

Apps
I’d like to say that this phone has decent app support. But it doesn’t. If you want a phone with lots of apps, get an Android device, or an iPhone. No one seems to make decent apps for the BB. Yes, theres apps, you can usually find something to make your BB do what you want it to, but it’s never a fit and polished as an app for more popular devices.

Text input
For a Blackberry, the text input on this phone is actually rather horrid. The Surepress screen, while comforting at first, becomes clumbsey, and actually slows you down. At the point where you’d be getting used to touch-screen input on another device, you find that now you’re held back by this damned press screen. I use the SureType input method. This is similar to the BlackBerry Pearl’s input method, where you’re presented with a qwerty format, where each key is shared by two letters. Its the closest thing to a qwerty keyboard that you can use with one hand. In landscape mode, you get a true qwerty, but its even slower than Suretype because of this damned surepress screen. On top of that, you don’t get very good feedback on what key you’re about to press.

Overall performance
The phone performs ok, but not for long. After a few days the phone starts to slow down, and eventually becomes so painful to use that it needs to be reset by pulling the battery.

On top of all that, the phone lacks one of the things that I think all smart phones on the market today should have. ActiveSync support. Any business phone should be able to talk to an ActiveSync server if it wants to be a contender. RIM doesn’t want you to have this feature, because they want you, or your company, to buy a BES. I purchased an app, which adds ActiveSync, but running it in the background eats up resources like you wouldn’t believe. If i run it, i get about 3 days between reboots on the phone. If I don’t run it, i get about a week.

All that being said, I intend to upgrade to a Droid device as soon as I can. …

I’m a few days late reporting this, because I’ve been busy, but on 8/13 my wife delivered our baby girl! A full account of this ordeal can be found here.

Yesterday I hopped on to Facebook to check in for the evening. I found a friend of mine had posted a link to a news article. His comments made me think (before i even read the article) that he was shocked at the content, and disagreed with it. After I read the article, and watched the attached broadcast news clip, I understood why.

The article, about a Hallmark graduation card, which offended the NAACP went on to describe how one of these greeting cards, with the cheapo internal speakers, that plays a greeting when you open it, was demeaning to black people, and the NAACP was making a stink over it. Hallmark, in order to avoid offending people (which is noble enough) decided to remove the card from shelves and apologized. They also stated that the card had been out on shelves for 3 years, and no one had complained thus far.

The complaint, to sum up for anyone who hasn’t read the article, was that the audio in the card used the term “black whores”, and this was obviously intended to stifle the aspirations of black women. I’ll say this. If the card ACTUALLY SAID THIS, I’d agree 100%, lead the charge against Hallmark for being such insensitive ass holes, and probably wouldn’t be writing this blog post…. Yea, so that’s what I’d be doing if there was a legitimate complaint there. The card didn’t say that however, it said “black holes”. The card was themed after the solar system, talking about how the graduate could take on anything. I wasn’t able to find a full transcript from the card, as that would give you, you know, CONTEXT, but here’s the offending snip.

And you black holes, you are so ominous. Watch your back

What are these people thinking? How could someone seriously believe that hallmark would really release a card with such an obviously racist, no, outright offensive, statement? Short answer? These people are LOOKING FOR A FIGHT. The race issue, in my opinion, is calming down. We have after all elected a mixed race president, and you know what, the man’s race never enters my mind. I disagree with his politics, but that’s how it should be, right? If you ask me, racism would slowly vanish if groups, and individuals would stop playing the victim and bringing it back to the forefront.

Let me show you where I’m coming from before I go down the rant highway. I’m in my early 30’s, I live in a nice neighbourhood, I have a wife, two dogs, and a kid on the way. I have a steady job, I work for what I have, and I like it that way. Yea, sure, everyone likes something for free, and I’m no exception, but I refuse to live off of someone else. I live within my means as best as I’m able. I’m not racist. Personally I think that the majority of our society have moved past race. Yea, it’s still out there, but it’s not like a minorities have to fight for the right to sit and eat at a restaurant, or get a job where they’re competing with white people.

In fact, it’s to the point where things are almost the opposite. It’s more difficult for me to live, as your average white guy, because I’m paying for government programs that help out minorities. Tuition, assistance, all sorts of leeway for anyone who’s not white. I’m starting to feel like I am the minority here!

This is what I mean when i say civil rights has failed. Not that the minorities and those with disabilities that this is meant to help are not getting help, but that they’re getting so much help that it’s actually hurting the rest of us.

…

A few months ago, I inherited a used server, and decided to install xen on it, and then create myself web and mail vm’s. This would then replace my current web server.

I installed a web vm, which works nicely, a second web vm for a friend, and finally, a Zimbra VM. The Zimbra vm was troublesome. My hardware was old, it wasn’t the best platform for virtualization, and zimbra seems unhappy in a xen VM.

After i finished migrating all of my data from my old web server to the new web vm, i decided to give zimbra a shot on metal. So i installed Zimbra on my old web server (after stripping out all of my web related accounts and services of course). Performance was better, and zimbra seemed happier. So i set about trying to find a nice clean way to move all of my zimbra data from the VM, to the new install.

Well, Open Source Edition does not include the nice backup features of Network Edition, which would have probably made this a lot easier. So I was left to find my own means. I found some talk on the zimbra forums about how this might be accomplished. Simply install zimbra on the target server, then copy /opt/zimbra from the source server, to the destination server. This depends on similar configurations on both ends though, and this did not end well for me. After hours of work (and hours of an scp copy) i ended up with a zimbra install that appeared to function, but postfix was hosed beyond belief. This seems at least partially due to the fact that my target system had a different hostname than the source system. This was almost a necessity for me, as my vm server, and my old web server are located in different datacenter’s, many many miles apart.

So i decided to give something else a shot. Let’s take zimbra’s ldap, mysql, and store, and just tear it out of the source and import it into the target. Here’s how i did it:

I started out with two servers. One the production server (source, “interchange”) and one the newly setup server (target, “stimpson”). On the target, i installed zimbra, fresh and clean. I did a few customizations during the install such as time zone, and alert e-mails, but nothing fancy.

First, you’ll want to stop your MTA on the source, so new mail coming in is queued at the sender. This way you’re sure not to miss any mail.

[zimbra@interchange ~]$ zmmtactl stop

Now, zimbra is still running, but it’s not accepting mail. Any mesages sent to our server should be queued at the sender, and re-tried later. Don’t leave your system in this state for too long however, as you have no idea how long the sending server will queue these messages. A few hours is probably a safe bet.

Now, backup all of your data.

ldap:
Zimbra, in zcs 6, includes a slapcat like script, for just this purpose.

[zimbra@interchange ~]$ /opt/zimbra/libexec/zmslapcat /tmp/foo

This places an ldap.bak file in /tmp/foo. This is an ldif file, you can rename it. I called mine interchange.ldif

Mysql:
I had to create a script for mysqldump, this was included in earlier versions of zimbra, but they removed it. I’m not sure why.
I took ~zimbra/bin/mysql and copied it to ~zimbra/bin/mysqldump, and then modified it such:

[zimbra@interchange ~]$ cat ~zimbra/bin/mysqldump 
#!/bin/bash
# 
# ***** BEGIN LICENSE BLOCK *****
# 
# Zimbra Collaboration Suite Server
# Copyright (C) 2004, 2005, 2006, 2007 Zimbra, Inc.
# 
# The contents of this file are subject to the Yahoo! Public License
# Version 1.0 ("License"); you may not use this file except in
# compliance with the License.  You may obtain a copy of the License at
# http://www.zimbra.com/license.
# 
# Software distributed under the License is distributed on an "AS IS"
# basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.
# 
# ***** END LICENSE BLOCK *****
# 

source `dirname $0`/zmshutil || exit 1
zmsetvars mysql_directory mysql_socket zimbra_mysql_user zimbra_mysql_password

exec ${mysql_directory}/bin/mysqldump -S ${mysql_socket} \
    -u ${zimbra_mysql_user} --password=${zimbra_mysql_password} "$@"

Then, just dump mysql, as you would on any mysql server, you do not need to specify login credentials however, you’ll see, that s included in the script.

[zimbra@interchange ~]$ mysqldump --all-databases > /tmp/foo/interchange.sql

Store:
The store is simple (this includes the index as well).

[zimbra@interchange ~]$ tar czf /tmp/foo/interchange_store.tar.gz /opt/zimbra/store /opt/zimbra/index

Now, however its convenient for you, get all of these files to the target server.

On the target, stop zimbra.

[zimbra@stimpson ~]$ zmcontrol stop
Host stimpson.undrground.org
	Stopping stats...Done.
	Stopping mta...Done.
	Stopping spell...Done.
	Stopping snmp...Done.
	Stopping archiving...Done.
	Stopping antivirus...Done.
	Stopping antispam...Done.
	Stopping imapproxy...Done.
	Stopping memcached...Done.
	Stopping mailbox...Done.
	Stopping logger...Done.
	Stopping ldap...Done.

Now, i had to do some modification to my data files. because of hostname changes and whatnot.

If you want to make a backup of /opt/zimbra in case something goes wrong, and you need to start with a fresh install, now’s the time to do it.

[root@stimpson /opt]# cp -rp zimbra/ zimbra.fresh

Ldap contains all of your account data, AND a lot of your configuration. I found it cleanest to strip all but the account data out. This was done by beheading the ldif of the first 1615 lines. You may need to look at your ldif and see if this is the right number for you, i found that an ldif i slapcatted from the target server (for comparison) had about 100 fewer lines that needed to be hacked off, so you’ll have to look. I found in the ldif file where the COS definitions started, and hacked off every thing before that. This worked mostly well, i did run into one issue with one domain. I’ll cover that later.

I also searched for any occurance of “interchange” and swapped it with “stimpson”.

The sql file we dumped also needs to be updated. A simple global search/replace for your old hostname, to your new hostname is all you need. In my case, i searched for interchange, and replaced it with stimpson.

Now, the real fun begins.

Import ldap:
Be sure zimbra is stopped (see above).
Import your ldif with the following command:

[zimbra@stimpson ~]$  ~zimbra/openldap-2.4.21.3z/sbin/slapadd -F ~zimbra/data/ldap/config -l interchange.ldif

I did not capture the output of this command, it returns a progress bar, and then a success message.

Mysql:
In order to import mysql, you’ll need mysql running. It’s safe to start up zimbra at this point, so go ahead.

[zimbra@stimpson ~]$ zmcontrol start
Host stimpson.undrground.org
	Starting ldap...Done.
	Starting logger...Done.
	Starting mailbox...Done.
	Starting antispam...Done.
	Starting antivirus...Done.
	Starting snmp...Done.
	Starting spell...Done.
	Starting mta...Done.
	Starting stats...Done.

You’ll need to grab some information from your source, and target servers now. Namely, the mysql password. It will be different from one server to the next, and once you import your mysql dump, it will be reset (unless you go into the .sql file, and remove the piece which resets it.. up to you).

[zimbra@interchange ~]$ zmlocalconfig -s | grep zimbra_mysql_password
zimbra_mysql_password = bob

[zimbra@stimpson ~]$ zmlocalconfig -s | grep zimbra_mysql_password
zimbra_mysql_password = joe

Yes, i’ve changed these passwords, theyre not actually bob and joe, i cant go giving my mysql password out to the world. What you’ll have here is some long, random, password. Make note of both of them, and which servers they belong to.

Once zimbra is started, mysql will also be running, so now we can import our database.

[zimbra@stimpson ~]$ mysql < interchange.sql

Once its imported, you'll find that you can no longer access mysql.

[zimbra@stimpson ~]$ mysql
ERROR 1045 (28000): Access denied for user 'zimbra'@'localhost' (using password: YES)

This is because zimbra@localhost's password has changed. The simplest way around this is to take zimbra's mysql script, and hack it up again.
So i took ~zimbra/bin/mysql and copied it to ~zimbra/bin/mysql.mod then made the following change

[[zimbra@stimpson ~]$ cat ~zimbra/bin/mysql.mod 
#!/bin/bash
# 
# ***** BEGIN LICENSE BLOCK *****
# Zimbra Collaboration Suite Server
# Copyright (C) 2004, 2005, 2006, 2007, 2009, 2010 Zimbra, Inc.
# 
# The contents of this file are subject to the Zimbra Public License
# Version 1.3 ("License"); you may not use this file except in
# compliance with the License.  You may obtain a copy of the License at
# http://www.zimbra.com/license.
# 
# Software distributed under the License is distributed on an "AS IS"
# basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.
# ***** END LICENSE BLOCK *****
# 

source `dirname $0`/zmshutil || exit 1
zmsetvars mysql_directory mysql_socket zimbra_mysql_user zimbra_mysql_password
		
exec ${mysql_directory}/bin/mysql -S ${mysql_socket} \
    -u ${zimbra_mysql_user} --password=bob "$@"

Notice that --password says bob now.

Now connect to mysql, and issue the following commands. This is to reset zimbra's mysql password to what it was before the import.

[zimbra@stimpson ~]$ mysql.mod
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 328
Server version: 5.0.90-log Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed

mysql> set password for 'zimbra'@'localhost' = password('joe');
Query OK, 0 rows affected (0.02 sec)

mysql> set password for 'zimbra'@'localhost.localdomain' = password('joe');
Query OK, 0 rows affected (0.00 sec)

mysql> set password for 'zimbra'@'%' = password('joe');
Query OK, 0 rows affected (0.00 sec)

mysql> quit
Bye

Now mysql should be accessible to zimbra again.

Now, the store, this is the easy part.

Stop zimbra, you may not really need to do this, but it feels cleaner to me.

Depending on the path included in your tar archive (remember, mine was /opt/zimbra/store) you'll want to change to the appropriate directory. Since mine includes the absolute path, i'm moving to /, as root.

[root@stimpson /]# tar xvzf interchange_store.tar.gz

Your store will decompress, in my case to /opt/zimbra/store

Once it's complete, reset perms on /opt/zimbra store so that its owned by zimbra:zimbra.

[root@stimpson /]# chown -R zimbra:zimbra /opt/zimbra/store /opt/zimbra/index

Done! Now you should be set to go, start up zimbra!

Now you should be able to re-direct DNS, so that your MX for your zimbra domains are pointing to the proper place. Also, whatever your webmail addresses are should be updated.

For good measure, i also chkconfig off'd zimbra on the source server.

Issues i had:
I mentioned earlier that one of my domains didnt transfer over properly. I have about a half a dozen domains hosted. One of them, which happens to be the first domain i setup, didnt come over completely. All of the mailboxes were there, and i could login to them via webmail,. The only time i saw an issue was when i viewed the domains in the zimbra admin, or if a piece of e-mail came in for one of the addresses at the broken domain. They'd get bounced.

It seems to me that the ldap data i imported must have been incomplete. I was able to fix it rather easily though.

If i used zmprov to view all domains, the domain was listed.

zmprov getalldomains

But, if i tried to view detail on the broken domain, it returned an error stating that the domain didnt exist.

zmprov getdomain brokendomain.com

I assumed that zimbra added some sanity checking into their zmprov utility, and thought i'd give a simple modification to the domain a shot, and see if this fixed anything.

I renamed the domain via zmprov, and then renamed it back to its original name.

zmprov -l renameDomain brokendomain.com brokeddomain.com

zmprov -l renameDomain borkeddomain.com brokendomain.com

This resolved the issue.

Update:
See my thread on zimbra's forums for another method. Looks like I may have been able to accomplish this using the "Copy zimbra's home dir" method if i'd know about zmsetservername!

Ah well, i like my method better anyway. 😛

Update:
I found that my zimbra documents were not working, probably due to the wiki account changing.

The fix is simple, once you track it down.
Find out what your global wiki account is, you can get it with the following command:

[zimbra@stimpson ~]$ zmprov gacf | grep zimbraNotebookAccount
zimbraNotebookAccount: wiki@stimpson.(domain.name)

(domain.name) is your actual top level domain, of course. this is generally the fqdn of the first domain you setup, which is generally the fqdn of your zimbra server.

Now do the following:

[zimbra@stimpson ~]$ zmprov in wiki@stimpson.(domain.name)
Initializing folders 
Creating folder Template
[zimbra@stimpson ~]$ zmprov ut -h stimpson.(domain.name) /opt/zimbra/wiki/Template/updating global wiki account wiki@stimpson.(domain.name)
Initializing...
Creating wiki page _PathBodyTemplate in folder Template
Creating wiki page _Index in folder Template
Creating wiki page _TocItemTemplate in folder Template
Creating wiki page _TitleBar in folder Template
Creating wiki page _PathItemTemplate in folder Template
Creating wiki page _VersionTitleBar in folder Template
Creating wiki page _Template in folder Template
Creating wiki page _VersionIndex in folder Template
Creating wiki page _TemplateStyles_unused in folder Template
Creating wiki page _SideBar in folder Template
Creating file document Logo.gif in folder Template
Creating wiki page _VersionTemplate in folder Template
Creating wiki page _TocVersionBodyTemplate in folder Template
Creating wiki page _TocVersionItemTemplate in folder Template
Creating wiki page _TemplateStyles in folder Template
Creating wiki page _Footer in folder Template
Creating wiki page _Header in folder Template
Creating wiki page _TocBodyTemplate in folder Template
Creating wiki page _PathSeparator in folder Template

...