This should be handled fine now by .dir-locals.el, so need to carry that stuff in every file.
This patch removes includes that are not used. The removals were found with include-what-you-use which checks if any of the symbols from a header is in use.
We still only produce on .so, but let's keep the sources separate to make things a bit less messy.