Background padding (inset drawable) not working as expected

Background padding (inset drawable) not working as expected

I want to achieve something like this:

As stated here InsetDrawables can be used to make backgrounds smaller than
the View that contains it. However when I do this all my child views
(ImageView, TextView) align with the bottom border of the inset
background. Any idea how I can achieve this effect?