# ipfs-search.com - Full Application Content ## Project Overview **Name:** ipfs-search.com **Tagline:** Search engine for the Interplanetary File System **Fund:** NGI0 Discovery **Period:** 2019-04 — 2022-10 **Grant Agreement No:** 825322 **URL:** https://nlnet.nl/project/IPFS-search/ **Website:** https://ipfs-search.com **Source Code:** https://github.com/ipfs-search/ipfs-search **Documentation:** https://ipfs-search.readthedocs.io **Blog:** https://blog.ipfs-search.com **Contact:** Matrix #ipfs-search:matrix.org --- ## Abstract ipfs-search.com is a Free and Open Source (FOSS) search engine for directories, documents, videos, music on the InterPlanetary Filesystem (IPFS), supporting the creation of a decentralized web where privacy is possible, censorship is difficult, and the internet can remain open to all. The ipfs-search command consists of two components: the crawler and the sniffer. The sniffer extracts hashes from the gossip between IPFS nodes. The crawler extracts data from the hashes and indexes them. Metadata and contents are extracted using ipfs-tika, searching is done using OpenSearch, queueing is done using RabbitMQ. The crawler is implemented in Go, the API and frontend are built using Node.js. --- ## Why does this actually matter to end users? On the web it can happen that someone who is hosting a small website that is valuable to you no longer wishes to operate it. Or is unable to bear the costs. Or is censored. She turns off the server without warning (or someone does it for her). And all the information ever submitted is lost. If that information is for instance not just some web app, but an original resource that is valuable from a personal, cultural or historical point of view, that feels like a waste. We often learn a lot while browsing through old books, brochures, news papers and advertisements - but also from resurfacing our own notes from the past. Surely, valuable institutions like the Internet Archive can have potentially archive anything. But they won't be able to capture everything, and furthermore an archive is a frozen state. What if this was actually a community resource that you as a user contributed a lot of time and content to, and have a stake in wanting to keep it alive? Wouldn't it be great if you as a contributor had the "source" all along, and were able to bring it back to life? IPFS, or the InterPlanetary File System, is different from the original web that assumes and thus assigns all the rights to a single server and owner. It is like a giant virtual storage cluster spread across the computers of all the people that use it. You can use it to publish things, such as web pages. One thing is different: as long as there are people that retain a copy of those pages, it will remain available. IPFS does not per se have a central place of storage, any copy is enough to give others access to this data by sharing links to the content. That means you as a user can help curate the cultural, social, technical heritage you care about. The approach is similar to torrenting, but IPFS offers more transparency and security since users can look at past versions of edited data and content is uniquely identified and permanently stored. One significant issue though for users is that conventional search engines at present do not engage with the InterPlanetary File System just yet. This is because it is challenging to work with such a distributed system, especially for a search engine. It is a constantly changing, complex and vibrant space, very much like the real world is. Easily finding things you need is vital to making any information system usable, especially if the system in question is huge (and "interplanetary" definitely has some ambitions there). This is where the project ipfs-search.com comes in. It aims to let users search the InterPlanetary File System through a regular search engine interface, that is nevertheless distributed over the IPFS-system itself. The actual indexing of files, which is automatically done by something called a crawler, can also be decentralized. This would ensure that search and discovery on IPFS would not become a single point of failure that can get taken down, attacked or censored. But actually belongs to and is operated by the same people that make the IPFS possible in the first place, namely its users. --- ## Technical Architecture ### Current Architecture The ipfs-search system consists of: - **Sniffer:** Extracts hashes from DHT gossip between IPFS nodes - **Crawler:** Extracts data from discovered hashes and indexes them - **ipfs-tika:** Used for metadata and content extraction - **OpenSearch:** Search engine backend (previously Elasticsearch) - **RabbitMQ:** Message queue for task distribution - **Go crawler:** Core crawling engine - **Node.js API and frontend:** Web interface ### IPFS Data Types The system handles various IPFS data types: - Files (regular content) - Directories (listings) - Raw data ### Documentation - Read the Docs: https://ipfs-search.readthedocs.io - Go documentation: https://pkg.go.dev/github.com/ipfs-search/ipfs-search - SwaggerHub OpenAPI: https://app.swaggerhub.com/apis-docs/ipfs-search/ipfs-search/ --- ## Roadmap: Towards Distributed Search ### Phase 1: Internet-facing Demo - Current IPFS Search architecture - Good enough indexing for initial functionality ### Phase 2: Technologies for Distributed Search - IPFS integration improvements - Overlay networks for search distribution - Parsing enhancements - Distributing the index on an IPFS Cluster - Querying the distributed index - Yggdrasil network integration - Testing and validation ### Phase 3: Moonshot - Fully Distributed Search - Provider nodes that wish to participate, parse and index only the files they have added to a dweb (DHT hashes) and that have world file permissions - Local index put on an IPFS cluster - Distributed query using the distributed index - Initial search: basic boolean search - Settings functionality for tuning - Extension-based feature additions ### Related Search Engine Comparison - **seekstorm (former FAROO):** Proprietary, English - **Seeks:** Open Source, English - **YaCy:** Fully distributed - **Noetic:** Searches IPFS content as indexed by conventional search engines (inactive) - **Trinity:** Promising but currently non-functional - **ipfsearch.xyz:** Searches decentrally but requires pre-compiled database - **dweb.page:** Uses IOTA distributed ledger --- ## Dependencies ### External - Go 1.19 - OpenSearch 2.3.x - RabbitMQ / AMQP server - NodeJS 9.x - IPFS 0.7 - Redis ### Internal - nsfw-server: https://github.com/ipfs-search/nsfw-server - ipfs-tika: https://github.com/ipfs-search/ipfs-tika --- ## Related Repositories - Frontend: https://github.com/ipfs-search/ipfs-search-frontend - Search API: https://github.com/ipfs-search/ipfs-search-api - Deployment: https://github.com/ipfs-search/ipfs-search-deployment - NSFW Server: https://github.com/ipfs-search/nsfw-server - IPFS Tika: https://github.com/ipfs-search/ipfs-tika --- ## Snapshots ipfs-search provides daily snapshots for all indexed data using OpenSearch snapshots. This allows the community to download and restore the complete search index. --- ## Building and Running ### Docker (Recommended) ```bash docker-compose up ``` ### Manual ```bash $ go get ./... $ make ``` ### Ansible Deployment Automated deployment on Ubuntu 16.04 machines via https://github.com/ipfs-search/ipfs-search-deployment --- ## Sponsors and Support - **NLnet Foundation:** Funded through NGI0 Discovery, EU Next Generation Internet programme - **RedPencil:** Hosting support for ipfs-search.com - **Open Collective:** Community funding --- ## Themes - NGI Zero Discovery - Verticals + Search --- ## Funded By This project was funded through the NGI0 Discovery Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG Communications Networks, Content and Technology under grant agreement No 825322.