In this paper, we address whether the growth of supermassive black-holes has kept pace with the process of galaxy assembly. For this purpose, we first searched the Hubble Ultra Deep Field (HUDF) for "tadpole galaxies", which have a knot at one end and an extended tail. They appear dynamically unrelaxed -- presumably early-stage mergers -- and make up ~6% of the field galaxy population. Their redshift distribution follows that of field galaxies, indicating that -- if tadpole galaxies are indeed dynamically young -- the process of galaxy assembly generally kept up with the reservoir of field galaxies as a function of epoch. Next, we present a search for HUDF objects with point-source components that are optically variable (at the >~3.0 sigma level) on timescales of weeks--months. Among 4644 objects to i_AB=28.0 mag (10 sigma), 45 have variable point-like components, which are likely weak AGN. About 1% of all field objects show variability for 0.1 < z < 4.5, and their redshift distribution is similar to that of field galaxies. Hence supermassive black-hole growth in weak AGN likely also kept up with the process of galaxy assembly. However, the faint AGN sample has almost no overlap with the tadpole sample, which was predicted by recent hydrodynamical numerical simulations. This suggests that tadpole galaxies are early-stage mergers, which likely preceded the ``turn-on'' of the AGN component and the onset of visible point-source variability by >~1 Gyr.