The adjective sense "being on the inside" is from 1610s, from the noun. It began to be used in slang c. 1900 in reference to the supposed real facts or situation that only an insider would know. Inside man is from 1911 (originally in reference to workers used by management to sniff out union activity); inside job "robbery, espionage, etc., committed by or with the help of a resident or servant of a place" is attested by 1887, American English (also, late 19c., early 20c., "indoors work").
The figurative inside track "advantage" (1854) however is a metaphor from horse racing (1830); inside lanes are shorter than the outer ones on a curved track. Adverbial use in American English inside of (in reference to time) is from 1839.