Backus was born in Philadelphia on December 3, 1924, and grew up in Wilmington, Delaware. He was a student at The Hill School in Pottstown, Pennsylvania, although he did not appear to be a diligent student. He abandoned his chemistry studies at the University of Virginia and enlisted in the United States Army. He started medical training at Haverford College and was diagnosed with a cranial bone tumor during an internship at a hospital, which was successfully removed; a plate was placed in his head, and he finished medical school after nine months and a subsequent operation to replace the plate with one of his own design.

He then relocated to New York City, where he began his career as a radio technician before becoming interested in mathematics. He earned a master's degree in mathematics from Columbia University in 1949 and joined IBM in 1950. He worked on the Selective Sequence Electronic Calculator (SSEC) for his first three years, and his first big effort was to create a software to compute Moon locations. Backus developed Speed coding, the first high-level language for an IBM computer, in 1953 to help with software development for the IBM 701 computer.

Backus organized a team in 1954 to design and build FORTRAN for the IBM 704 computer, as programming was extremely difficult at the time. FORTRAN was the first widely used high-level programming language. Backus was also a member of the international committees that created ALGOL 58 and ALGOL 60, which swiftly became the de facto global standard for publishing algorithms. In the UNESCO report on ALGOL 58, Backus created the Backus–Naur form (BNF). It was a significant step in the development of compilers since it was a formal notation that could represent any context-free programming language.

Later, he studied the function-level programming paradigm and presented his results in the Turing Award lecture "Can Programming Be Liberated from the von Neumann Style?" in 1977. Backus received the W. W. McDowell Award from the IEEE in 1967 for developing FORTRAN. In 1975, he was awarded the National Medal of Science and in 1977, the ACM Turing Award for "profound, influential, and lasting contributions to the design of practical high-level programming systems, particularly through his work on FORTRAN, and for publication of formal procedures for the specification of programming languages." He retired in 1991 and died on March 17, 2007, at his home in Ashland, Oregon.