Welcome to FSCrawler for Elasticsearch

This crawler helps to index binary documents such as PDF, Open Office, MS Office.

Main features:

• Local file system (or a mounted drive) crawling and index new files, update existing ones and removes old ones.
• Remote file system over SSH/FTP crawling.
• REST interface to let you "upload" your binary documents to elasticsearch.

Latest versions

Current "most stable" versions are:

Elasticsearch	FSCrawler	Released	Docs
7.x, 8.x, 9.x	3.0-SNAPSHOT		3.0-SNAPSHOT

Quick start

Run Elasticsearch with start-local:

# Start Elasticsearch and Kibana
curl -fsSL https://elastic.co/start-local | sh
# Get the generated API key (you will need it for FSCrawler)
source elastic-start-local/.env

Run FSCrawler with Docker:

docker pull dadoonet/fscrawler
docker run -it --rm \
  --add-host=host.docker.internal:host-gateway \
  -v ~/.fscrawler:/root/.fscrawler \
  -v $(pwd)/resumes:/tmp/es:ro \
  -e FSCRAWLER_ELASTICSEARCH_URLS=http://host.docker.internal:9200 \
  -e FSCRAWLER_ELASTICSEARCH_API_KEY="${ES_LOCAL_API_KEY}" \
  -e FS_JAVA_OPTS="-DLOG_LEVEL=debug" \
  dadoonet/fscrawler

Then open Kibana and watch for your documents coming to the fscrawler alias:

FROM fscrawler 
| STATS numDocs = COUNT(*)

Or search for some text:

FROM fscrawler
| WHERE content : "David"

Or count by file.content_type:

FROM fscrawler 
| STATS numDocs = COUNT(*) BY file.content_type

Note:

• ~/resumes contains the documents you want to index
• Job settings will be stored in ~/.fscrawler/fscrawler/_settings.yaml

Read the documentation for more details and specifically the tutorial page.

Project information

Stats

Version in preparation

Latest release

Build & quality

License

Read more about the Apache2 License.

Thanks

Thanks to JetBrains for the IntelliJ IDEA License! The best IDE out there!

Thanks to SonarCloud for the free analysis! You guys rock!