LLMs and the web

Early web browsers (e.g. Mosaic in the 1990s and its successors) relied on centralised servers that pre-processed (“crawled”) web pages to identify key terms, words and groups of word. The servers would then add these terms to an index optimised for quick lookup, linking the terms to the URLs of the pages in which these…More