Spain is the first olive oil producer worldwide (representing around 45%), with more than 80% of its agricultural area dedicated to this crop. The goal of this study is to assess the environmental impact of the Spanish virgin olive oil production, attending to the farming and industrial phases. A Life Cycle Assessment (LCA) is conducted for 5 harvests in approximately 4.000 ha of olive grove in Jaen, the largest producer region of Spain. The type of farming is the most representative in Spain: conventional (non-organic), with a medium-low slope, extensive (100–150 trees per ha) and around 60% dryland orchards. The industrial phase is based on 2-phase extraction process and olive pomace valorization. The functional unit (FU) chosen for the comparative analysis is 1 kg of unpacked virgin olive oil under a perspective from “cradle to gate”. One of the most representative categories, climate change, places the average environmental impact in 2.43 kg CO2 eq/kg, while the range is between 1.93 and 3.00 kg CO2 eq/kg depending on the harvest. Huge differences between values are observed in the farming phase and they are mostly caused by the virgin olive oil yield of every harvest. The carbon sequestration analyses and the impact produced per FU ranges from 43.78% (for 15/16 harvest) to 46.36% (for 17/18). The impact hotspots detected in the farming phase, in terms of climate change potential, are the categories of plant protection products and herbicides (24.11% of the average value) and fertilizers (20.01%). However, the environmental impact of the industrial phase is relatively constant, with a value of 0.547–0.554 kg of CO2 equivalent. The 80.74% is caused by pomace valorization, which translated to the average whole value, represents the 18.56%.