You need to strengthen the surrounding muscles to take some of the abuse and to provide stabilization for the muscles involved in riding. Additionally, you need to start a stretching routine to increase flexibility and do some type of warm-up prior to riding. I would also maybe recommend getting something like a golf ball and putting the ball between your anterior arm where the pain is and the wall, and then rolling it with pressure.
Simply riding more is not going to do you any favors. The reason you feel pain is most likely because when you first started riding, you would slightly tear the muscles and tendons. These heal at different rates, but will result in small amounts of scar tissue. So, you are either feeling pain because you have a build up of scar tissue that is inflexible and simply tears again, or because previous tears are still around. It is possible that both of these are occurring because tendon tears take a long time to heal properly because of less blood flow (slower healing).