Particle Swarm Optimization (PSO), whose concept has been established as a simulation of a simplified social milieu, is known as one of the most useful optimization methods for solving non-convex continuous optimization problems. This paper describes a new learning algorithm to simultaneously adjust connection weights included in neural networks and some user-specified parameters included in units. According to the proposed algorithm, it is possible to improve the learning properties of the neural networks, e.g., the learning cost and/or adaptability. The behavior of the proposed algorithm is examined on a numerical simulation example.