Custom blocks are the placeable machines (hoppers, collectors, special chests, chunk loaders) that other modes sell and hand out as rewards. In Prison none of them are currently obtainable: no shop entry, crate or reward gives them out.
Automation in Prison comes from the pickaxe, its enchantments and robots instead; see Prison Enchants and Minions. Special items are covered on Custom Items.