However some bacteria do fossilise, leaving behind perminant records of their existance. One of the ones that does it the best is cyanobacteria, which can form fossils in two ways. Firstly by forming little calcified shells around themselves as a product of increasing the dissolved carbon dioxide levels inside the cell:
Cyanobacteria fossils - from Berkeley university page
Secondly cyanobacteria can group together with algae to form large layered mat-like structures called stromatolites. Slicing these very thinly reveals tiny cyanobacteria fossils, caught between the layers. There are stromatolytes in Australia that have been alive and growing (very slowly) for millions of years, despite looking faintly uninspiring.
Maybe not quite as impressive as dinosaurs...
While no other bacteria form fossils as nicely preserved as the cyanobacteria, they are capable of leaving behind visible remnants of their existence. The polypeptides of the bacterial cell wall (along with cytoplasm and some secreted lipids) can, under certain conditions, act as nucleation sites for minerals. This eventually leads to the organic cell being replaced by a little mineral cast of the bacteria. There is the possibility for quite a few artifacts with this (artifact being the scientific word for "result caused by the preservation process rather than the bacteria") the most common one being the creation of an artificial nucleus structure. As the bacteria degrades the cytoplasm tends to clump, and the crystallisation of minerals around a cytoplasm clot can in some cases create a structure that looks similar to the structure formed by a nucleus.
Endolithic bacteria that live in rocks can leave behind tiny canals in the rock surface that they bore into. These are a lot harder to find and interpretation is usually helped by the discovery of nearby alive endolithic bacteria. Bacteria have also been found trapped and mumified inside tree resin, a la the Jurassic park mosquito.
Once you start getting larger organisms the bacteria have a brand new niche to exploit. Fossilised bones sometimes show the results of a bacterial infection; while the bacteria themselves are not being preserved their presence is still seen in the fossil record.
Unfortunately although these glimpses are really interesting their also kind of frustrating from a biochemical point of view. They provide clues as to the lifestyles and processes within the bacteria but they are such tiny clues. Past biochemical processes are more often found inside the actual bacteria, by looking at clues in the genome and the genomes of related bacteria, than they are in the remains the bacteria leave behind.
Perri, E., & Tucker, M. (2007). Bacterial fossils and microbial dolomite in Triassic stromatolites Geology, 35 (3) DOI: 10.1130/G23354A.1
Westall, F. (2001). Early Archean fossil bacteria and biofilms in hydrothermally-influenced sediments from the Barberton greenstone belt, South Africa Precambrian Research, 106 (1-2), 93-116 DOI: 10.1016/S0301-9268(00)00127-3