A study of finite-size effects is carried out for hadron masses in the quenched simulation of lattice QCD using the Kogut-Susskind quark action. It is found that finite-size effects for quenched QCD are much smaller than those for full QCD, when hadron masses for the two cases are compared at the same physical lattice size and lattice spacing. Based on an extensive study of the boundary condition dependence of hadron masses we ascribe the origin of the difference to a partial cancellation of the finite-size effects among the Z(3)-related gauge configurations in quenched QCD; such a cancellation does not take place in full QCD due to Z(3) breaking effects of dynamical quarks. However, this does not mean finite-size errors are negligible in quenched QCD for lattice sizes of 2 to 3 fm used in current simulations; a still significant finite-size shift of hadron masses, especially of the nucleon mass, would pose a serious hindrance to obtaining the hadron mass spectrum at the few percent level aimed at in current quenched QCD simulations.