Search

Learn

• Build Yourself a Mini Search Engine. 2018.*
  • Discusses Nutch, Solr, crawling and indexing, ranking, reranking, and so on. Concise.
• Building an Open Source Meta-Search Engine. 2018.
  • A short paper as well as a code repo for Helios, the author’s implementation.
• I Built My Own Web Search Engine– Here’s How. 2018.
  • Author builds a small search engine using a Python stack including Flask and Scrapy.
• Let’s Build a Search Engine. Anvil.
  • Part 1: How to Build a Search Engine.
  • Part 2: How PageRank Works.
  • Part 3: How to Run Queries.
• Solr vs. Elasticsearch: Performance Differences & More. 2019.
  • sematext appears to be an industry expert on both Solr and Elasticsearch.
• Challenges in Running a Commercial Web Search Engine.
  • PowerPoint by Amit Singhal on Google.
• Metasearch Engine. 2013.
  • Written by Siobhan Grayson as part of Masters in Computer Science at UCD Dublin.
• How to Build a Search Engine. PacktPub, 2016.*

Engines

• Apache Lucene – open source, search library (Lucene core).
  • Welcome to Lucene Tutorial.
• Apache Solr – open source, enterprise search built on Apache Lucene.
  • We have an entire page dedicated to Apache Solr.
  • Official Reference Guide.
  • Sematext’s Get Started with Apache Solr.
  • Welcome to Solr Tutorial.
• Elasticsearch – open source, freemium, built on Apache Lucene.
• Carrot2 – open source, clusters results for smaller collections.
• Flexsearch – open source, F#/.NET, built on Apache Lucene.
• Sphinx – open source, C++ search engine, has been used on some large sites (Craigslist, Boardreader).
• SeekQuarry/Yioop – An open source PHP search engine.
• Searx – A spiritual successor to Seeks.
• Terrier – For large document collections, written in Java, open source.
• Riot – An open source, distributed search engine written in Go.
• Yacy – decentralized, open source, P2P.
• Gigablast – open source search engine/crawler written in C/C++.
• Toshi – search engine built on Tantivy in Rust.
• Susper – distributed, built on YaCy and Solr.
• Lemur Project: Indri – offers a number of different search related projects, Indri is the engine which can handle up to 500 million documents.
• Manticore Search – open source, for big data and stream filtering.
• Open Semantic Search – built upon other OSS technology like Solr, Elasticsearch, and Tika.
• Infinity Search – Open source, meta search.
  • As of 2/1/21 a search for “civil war” and “python” using the default meta search options did not return outstanding results.

Other Resources

• The Wayback Archive’s GitHub Repositories.
• UK Web Archive GitHub Repositories.

Related Books

• W. Bruce Croft, Donald Metzler, Trevor Strohman. Search Engines: Information Retrieval in Practice. 2015.
  • Available for free from the University of Massachusetts.
• Ian H. Witten, Marco Gori, Teresa Numerico. Web Dragons. Morgan Kaufmann, 7/2010. ISBN: 9780080469096.
• Christopher D. Manning, Prabhakar Raghavan, Hinrich Schutze. Introduction to Information Retrieval. Cambridge University Press, 2008.
  • Available for free on Stanford’s website.