Disable ido-ubiquitous in "read-buffer".
This is the function used by "switch-to-buffer", which is the function
originally bound to C-x C-b before ido is enabled. It is used as a
non-ido fallback when pressing C-b inside "ido-switch-buffer", so
allowing ido to work in it doesn't make sense, even though technically
it works just fine.