You do have to be very careful, because the inner frame is plastic, but it should come off. Slide the table knife (or something similar) in every inch or so and pull the panel out.
This is the way to do it, promise
I've replaced dozens of handles myself.