I'm confused by the lens name in your signature.
I'm not aware of a 75-300mm f/4-5.6 IS III. Is it the older IS USM model or the III USM? I'll take a guess and say the III USM given its price.
If that is the case, the only downside that I can think of is the lack of IS. To give an impression of speed, you would need to knock down your shutter speed so there is a chance you could have more motion blur from camera shake than you'd find desirable. However, if you have steady hands or can use a monopod/tripod, camera shake could cease to be of concern.
DSLRs: Canon EOS 7D Mark II, Canon EOS 70D
CSC: Canon EOS M3
Lenses: Canon EF-S 15-85mm f/3.5-5.6 IS USM, EF 50mm f/1.8 II, EF 100-400mm f/4.5-5.6L IS USM, Canon EF-M 18-55mm f/3.5-5.6 IS STM, Canon EF-M 22mm f/2.0 STM