Using the kappa-symmetric action for a D3-brane, we study the interaction between its world-volume fermions and a bosonic type IIB supergravity background preserving 4-dimensional Lorentz invariance. We find that the renormalizable terms in the action include only coupling between the fermions and the 3-form flux in the combination *G_3-iG_3, which is zero for a class of supersymmetric and nonsupersymmetric solutions. We also find the magnetic and electric dipole moments for the fermions, which are proportional to the derivative of the dilaton-axion. We show that different gauges to fix the kappa-symmetry give the same interaction terms, and prove that these terms are also SL(2,R) self-dual. We interpret our results in terms of N=1 supersymmetric gauge theory on the D-brane.