We have investigated the effect of pressure on the magnetic and structural properties of the frustrated heavy-fermion antiferromagnet YbAgGe in an extended pressure range up to 20 GPa using electrical resistivity and x-ray diffraction, respectively. We find that with increasing pressure, the magnetic ordering temperature (Tm) first increases rapidly, passes through a maximum (Tm=5.4 K at 6.8 GPa), and then drops toward zero for p above 16 GPa, while the structure remains unchanged up to about 20 GPa. We attribute the anomalous pressure dependence of Tm to a complex interplay between the Kondo effect, Ruderman-Kittel-Kasuya-Yosida interactions and the geometrical frustration of the magnetic interactions of the Yb moments in the Kagomé-latticelike structure of YbAgGe. We further discuss the competition between geometrical frustration and long-range magnetic order together with the possible occurrence of a pressure-induced second quantum critical point in YbAgGe.