When gravity can be neglected, spacetime is represented by the Minkowski space, which has metric and affine structures. According to the gauge `philosophy', we can read off the gravitational properties of matter by studying how matter-we will take a Dirac field as an example-couples in a non-inertial frame to the geometric structures of Minkowski space. The result is that the energy-momentum of matter couples to the coframe and the spin of matter to the Lorentz connection. If we relax the integrability conditions, which these potentials obey in Minkowski space, and drop the antisymmetry of the connection, we arrive at a metric-affine spacetime (including nonmetricity, torsion, and curvature) with the sources energy-momentum and hypermomentum. The weak gravitational potential, the coframe, is of the Newton-Einstein type, whereas the hypothetical strong gravitational potential is of Yang-Mills type and is possibly closely tied to matter. - We develop a Lagrange-Noether formalism and derive the field equations of gravity. We show that this general framework of MAG embodies various gravitational theories: The Poincare gauge theory (hypothetical, includes strong gravity), the Einstein-Cartan theory (viable theory, competes with general relativity, weak gravity plus spin-contact interaction), the teleparallel equivalent to general relativity (translational gauge theory, and general relativity itself.