1680s, pimiento (modern form from 1718), from Spanish pimiento "green or red pepper," also pimienta "black pepper," from Late Latin pigmenta, plural of pigmentum "vegetable juice," from Latin pigmentum "pigment" (see pigment (n.)). So called because it added a dash of color to food or drink.
[I]n med.L. spiced drink, hence spice, pepper (generally), Sp. pimiento, Fr. piment are applied to Cayenne or Guinea pepper, capsicum; in Eng. the name has passed to allspice or Jamaica pepper. [OED]
The piece of red sweet pepper stuffed in a pitted olive so called from 1918, earlier pimiento (1901), from Spanish. French piment is from Spanish.