I can see how this problem works in every respect except for one, which
is part (a). I assume that dm/M would be equal to the area of the blue
shaded disc divided by the area of the entire surface of the disc.
Hence:
dm/M = [(r+dr)^2 - r^2]/R^2
which reduces to: (2rdr+dr^2)/R^2, but the answer in the back is just
2rdr/R^2. What am I missing?
Terms that are second order (i.e. squared) or higher in infinitesimal quantities (such as dr) can be dropped when lower-order terms appear. In other words, (dr)^2 is always much much much less than rdr. Alternatively, you can think about area of the blue shaded region as having a circumferential 'length' of 2*pi*r, and a width of dr. The area is then 2*pi*r*dr.