Figurative Language with Brick by Brick