half transparent Matte Object

You can use the OccludedLayer render element to get the particles that are behind a matte object. Then in your compositing tool of choice, merge the normal render output over the geometry over the OccludedLayer output.

There is no way to get this to work perfectly unfortunately since this is a fundamental problem with the whole, matting/hold-out method.