See previous devlog post Ambient occlusions optimizations and comment I made:

"In implementing this optimization, I realized the voxel-level ambient occlusion could also be sped up even more if I keep bits in each voxel indicating if a given voxel face (same x, y and/or z plane) and its surrounding face plane neighbors are air voxels.  If the surrounding voxels for a given face are all air voxels, then the face doesn't need to be shaded by voxel-level ambient occlusion.  I'm working on this and it should yield even greater performance gains than the voxel-level ambient occlusion optimizations described above yielded."

I've got this working and frame rates with ambient occlusion are now at nearly 50% faster (mileage may vary depending on scene and view) from before I started optimizing ambient occlusion.  This latest optimization is not as much of a performance gain (+25%) as I was hoping for, but I'll take it as a win!

Here's a screen shot: