TO REBASE CHANGELOG.md, USAGE.md

Add implicit AND booleans at the end.

This fixes a number of issues in LooqsQuery:

(1) A query like a b c p:(something) would fail, because
a b c get merged into one word. This happens at the end.

lonewords are special and do not become a token immediatly. So previous
logic to add implicit ANDs does not apply.

(2) Negations were also broken with lonewords.

(3) The TokenType enum fields were too narrow to be useful for the bitmask

Independent of that, add support for 'limit:'

CHANGELOG.md

@@ -0,0 +1,22 @@
+# looqs: Release notes
+
+## 2022-06-XX - v0.1
+The first release comes with basic functionality. It's a start that can be considered useful to some degree.
+
+looqs is still at an early stage and may exhibit some weirdness and contain bugs.
+
+Tested architectures: amd64.
+
+CHANGES: 
+- CLI command "looqs" to add/update/delete and search 
+- GUI: "looqs-gui" to search, render previews, and add files to index 
+- General: Add multi-threaded indexing of all files (paths, mtime)
+- General: Generate sqlite based full-text search index for: .pdf,.odt,.ods, text files
+- General: Sandboxed content processing
+- GUI: Sandboxed IPC sub-process to render previews.
+- GUI: Add previews for pdf: Render the page the search keywords were found. Highlight the keywords when rendering the page. 
+- GUI: Add previews for plaintext files: Extract snippets. Highlight the keywords when rendering the page. 
+- General: Add basic filters for query.
+- Add packages: Ubuntu 22.04
+
+Thanks to all those who provided feedback (and endured bugs) at various stages. You know who you are, thx!

README.md

@@ -1,9 +1,11 @@
-# looqs - FTS for the Linux desktop with previews for search results
-looqs creates a full text search index for your files. It allows you to look at previews where your
+# looqs - Full-text search with previews for your files
+looqs is a tool that creates a full-text search index for your files. It allows you to look at previews where your
 search terms have been found, as shown in the screenshots below.
 
 
 ## Screenshots
+The screenshots in this section may occasionally be slightly outdated, but they are usually recent enough to get an overall impression of the current state.
+
 ### List
 ![Screenshot looqs results](https://garage.quitesimple.org/assets/looqs/opearting_systems_looqs.png)
 
@@ -15,7 +17,7 @@ search terms have been found, as shown in the screenshots below.
 ## Current status
 Last version: 2022-0X-XX, v0.1
 
-Please see [Changelog](CHANGELOG.md) for a human readable list of changes. 
+Please see [Changelog](CHANGELOG.md) for a human readable list of changes.
 
 
 ## Goals and principles
@@ -29,11 +31,13 @@ Please see [Changelog](CHANGELOG.md) for a human readable list of changes.
 Linux (on amd64) is currently the main focus. Currently, I don't plan on supporting anything else and the sandboxing architecture does not make it likely. I suppose a version without sandboxing might be conceivable for other platforms, but I have no plans or resources to actively target anything but Linux at this point.
 
 ### Licence
-GPLv3. 
+GPLv3.
 
 ### Contributing
-Fow now, github issues and pull-requests are preferred, but you can also just email
-your patches or issues to : looqs at quitesimple.org
+For now, github issues and pull-requests are preferred, but you can also just email
+your patches or issues to: looqs at quitesimple.org
+
+Please target the 'dev' branch in your pull request. 
 
 
 ## Build
@@ -47,10 +51,11 @@ qmake
 make
 ```
 
+The GUI is located in `gui/looqs-gui`, the binary for the CLI is in `cli/looqs`
 
 
 ## Documentation
-Please see [Usage.md](USAGE.md) for the user manual. 
+Please see [USAGE.md](USAGE.md) for the user manual.
 
 ## Packages
 Coming soon™

USAGE.md

@@ -0,0 +1,176 @@
+# looqs - User guide
+This document is still work in progress.
+
+## General points
+Please consult the [README](README.md) for a description of what looqs is and on how to obtain it.
+
+looqs is still at an early stage and may exhibit some weirdness and contain bugs.
+
+## Current Limitations and things to know
+You should be aware of the following:
+
+- It may seem natural, but the GUI and CLI operate on the same database, so if you add files using the CLI, the GUI will search them too.
+
+- If a file is listed in the "Search results" tab, it does not imply that a preview will be available in the "Previews" tab, as looqs can search more file formats than it can generate previews for currently.
+
+- Database paths are stored inefficiently, not deduplicated to simplify queries. This may add up quickly. Also, each PDF text is stored twice. Each page separately + the whole document to simplify queries.
+To give you some idea: At the time this section was written, 167874 files were in my index. A FTS index was built for 14280 of those, of which 4146 were PDF documents. The PDFs take around 10GB storage space on the filesystem. All files for which an FTS has been built are around 7GB in size on the filesystem. The looqs database had a size of 1.6 GB.
+
+- Existing files are considered modified when the mtime has changed. looqs currently does not check whether the content
+has changed. 
+
+## Config
+The config file is in `$HOME/.config/quitesimple.org/looqs.conf`. It will be created on first execution of the CLI or GUI
+interface. The GUI has a menu entry to quickly open this config file. This is to be considered temporary and will be removed once the GUI itself can edit all settings.
+
+Database default path: `$HOME/.local/share/quitesimple.org/looqs/looqs.sqlite`. If this does not work for
+you, move it and adjust adjust the path in the config file.
+
+
+## GUI
+The GUI is minimal at this point. Depending on what you want to do, you may need to open the config file and change the settings there. Chances are that you may not need to do that. 
+
+### First run
+You will be presented with an empty list. Go to the **"Index"** tab, add some directories and click **"Start indexing"**.
+
+For large directories the progress bar is essentially just decoration. As long as you see the counters
+increase, everything is fine even if it seems the progress bar is stuck.
+
+The indexing can be stopped. If you run it again you do not start from scratch, because looqs knows
+which files have been modified or not since they have been added to the index. Thus, files will
+only be reprocedded when necessary. Note that cancellation itself may take a moment as files finish processing. 
+
+### Search
+The text field at the top is where you type your query. It can be selected quickly using **CTRL + L**. Filters are avalable,
+see this document at the end. By default, both the full path and the content are searched. Path names take precedence.
+
+### Configuring PDF viewer
+It's most convenient if, when you click on a preview, the PDF reader opens the page you clicked. For that, looqs needs to know which viewer you want to launch.
+
+It tries to auto detect some common viewers. You must set the value of the ```pdfviewer=``` config entry yourself if it doesn't do something you
+like, such as not opening your favorite viewer. In the command line options, "%f" represents the filepath, "%p" the page number.
+
+### Preview tab
+The preview tab shows previews. It marks your search keywords too. Click on a preview to open the file.
+A right click on a preview allows you to copy the file path, or to open the containing folder. Hovering tells you which file the preview originates from.
+
+### Syncing index
+Over time, files get deleted or their content changes. Go to **looqs** -> **Sync index**. looqs will reindex the content of files which have been changed. Files that cannot be found anymore will be removed from the index.
+
+Reindexing a path using the "Index" tab will index new files and update existing ones. Currently however, this does not deal with deleted files.
+
+I recommend doing a sync from time to time.
+
+## CLI
+The CLI command "looqs" comes with helptext. This documentation is incomplete at the moment.
+
+### First run
+There is no point in using the "search" command on the first run. Add some files if not done so already.
+
+### Adding files
+To add files to the index, run ```looqs add [path]```, where 'path' can be a directory or a single file.
+If the path is a directory, the directory will be recursively descended, and all files in there added.
+
+"Skipped" implies the file has not been changed since it has been added to the index. If it has changed, the index content will be updated.
+
+### Searching files
+Of course the CLI will not render any previews, but it can show you the paths where search results
+have been found.
+
+```
+looqs search [terms...]
+```
+
+There is an implicit "AND" condition, meaning if you search for "photo" and "mountain", only paths
+will be shown containing both terms, but not either alone.
+
+### Deletion and Fixing Out of sync index
+You sometimes delete files, to get rid of those from the index too, run:
+
+```
+looqs delete --deleted --dry-run
+```
+
+This commands lists all files which are indexed, but which cannot be found anymore.
+
+Remove them using:
+```
+looqs delete --deleted --verbose
+```
+
+You can also delete by pattern:
+
+```
+looqs delete --pattern '*.java'
+```
+
+Delete never removes anything from the file system, it only operates on the database.
+
+The equivalent of the GUI sync command is:
+```
+looks update -v --continue --delete
+```
+
+
+### Updating files
+The content and metadata index for files can be updated:
+
+```
+looqs update -n
+```
+
+Those files still exist, but the content that has been indexed it out of date. This can be corrected with
+
+```
+looqs update
+```
+
+This will not add new files, you must run ```looqs add``` for this. For this reason, most users
+will probably seldomly use the 'update' command alone.
+
+
+## Tips
+
+### Keeping index up to date
+The most obvious way is to use the GUI and add your favorite paths in the "Index" tab. Then occasionally, just rescan. This works for me personally, looqs quickly picks up new files. This however may not be good enough for some users.
+
+Some users may prefer setting up cronjobs or wire up the CLI interface with file system monitoring tools such as [adhocify](https://github.com/quitesimpleorg/adhocify).
+
+### lh shell alias
+If you are in a shell and you know your file is somewhere in your current directory or its subdirs, and those
+are indexed by looqs, you may find the lh (look here) alias useful:
+
+```
+alias lh='looqs search $(pwd)'
+```
+
+So typing "lh recipes" searchs the current dir and its subdirs for a file containing 'recipes'.
+
+## Query syntax / Search filters
+A number of search filters are available.
+
+| Filter (long) |  Filter (short) | Explanation |
+| ----------- | ----------- |----------- |
+| path.contains:(term) | p:(term) | Pretty much a SQL LIKE '%term%' conditions, just searches the path string |
+| path.ends:(term) | pe:(term) | Filters path ending with the specified term, e. g.: pe:(.ogg) |
+| path.begins:(term) | pb:(term) |  Filters path beginning with the specified term |
+| contains:(terms) | c:(terms) | ull-text search, also understands quotes | 
+
+Filters can be combined. The booleans AND and OR are supported. Negations can be applied too, except for c:(). 
+The AND boolean is implicit and thus entering it strictly optional. 
+
+Examples:
+
+| Query | Explanation |
+| ----------- | ----------- |
+|pe:(.ogg) p:(marley)| Finds paths that end with .ogg and contain 'marley' (case-insensitive)
+|p:(slides) support vector machine           |Performs a content search for 'support vector machine' in all paths containing 'slides'|
+|p:(notes) (pe:(odt) OR pe:(docx))          |Finds files such as notes.docx, notes.odt but also any .docs and .odt when the path contains the string 'notes'|
+|memcpy !(pe:(.c) OR pe:(.cpp))| Performs a FTS search for 'memcpy' but excludes .cpp and .c files. 
+|c:("I think, therefore")|Performs a FTS search for the phrase "I think, therefore". 
+
+
+
+
+
+

cli/commandadd.cpp

@@ -40,7 +40,6 @@ int CommandAdd::handle(QStringList arguments)
 						"Continue adding files, don't exit on first error. If this option is not given, looqs will "
 						"exit asap, but it's possible that a few files will still be processed. "
 						"Set -t 1 to avoid this behavior, but processing will be slower. "},
-					   {{"a", "all"}, "On error, no files should be added, even already processed ones"},
 					   {{"v", "verbose"}, "Print skipped and added files"},
 					   {{"t", "threads"}, "Number of threads to use.", "threads"}});
 
@@ -50,10 +49,6 @@ int CommandAdd::handle(QStringList arguments)
 	parser.process(arguments);
 	bool keepGoing = parser.isSet("continue");
 	bool verbose = parser.isSet("verbose");
-	if(parser.isSet("all"))
-	{
-		throw LooqsGeneralException("To be implemented");
-	}
 	if(parser.isSet("threads"))
 	{
 		QString threadsCount = parser.value("threads");

cli/commandlist.cpp

@@ -7,29 +7,31 @@ int CommandList::handle(QStringList arguments)
 {
 	QCommandLineParser parser;
 	parser.addOptions({
-		{{"r", "reverse"}, "Print most-recent changed files first"},
-		{{"c", "count"}, "Counts the number of paths listed"},
-		{"pattern", "Only list files from index matching the pattern, e. g. */.git/*", "pattern"},
+		{"pattern", "Only list files from index matching the pattern, e. g. '*.txt'", "pattern"},
 	});
 
 	parser.addHelpOption();
 	parser.addPositionalArgument("list", "Lists paths in the index", "list [options]");
 
 	parser.process(arguments);
-	bool reverse = parser.isSet("reverse");
-	if(reverse)
+
+	QString pattern = parser.value("pattern");
+
+	QVector<FileData> results;
+
+	int offset = 0;
+	int limit = 1000;
+
+	auto resultscount = dbService->getFiles(results, pattern, offset, limit);
+	while(resultscount > 0)
 	{
-		throw LooqsGeneralException("Reverse option to be implemented");
+		for(FileData &fileData : results)
+		{
+			Logger::info() << fileData.absPath << Qt::endl;
+		}
+		offset += limit;
+		results.clear();
+		resultscount = dbService->getFiles(results, pattern, offset, limit);
 	}
-
-	QStringList files = parser.positionalArguments();
-	QString queryStrings = files.join(' ');
-	auto results = dbService->search(LooqsQuery::build(queryStrings, TokenType::FILTER_PATH_CONTAINS, false));
-
-	for(SearchResult &result : results)
-	{
-		Logger::info() << result.fileData.absPath << Qt::endl;
-	}
-
 	return 0;
 }

cli/commandlist.h

@@ -1,5 +1,5 @@
-#ifndef COMMANDSEARCH_H
-#define COMMANDSEARCH_H
+#ifndef COMMANDLIST_H
+#define COMMANDLIST_H
 #include "command.h"
 #include "../shared/sqlitesearch.h"
 
@@ -11,4 +11,4 @@ class CommandList : public Command
 	int handle(QStringList arguments) override;
 };
 
-#endif // COMMANDSEARCH_H
+#endif // COMMANDLIST_H

cli/commandsearch.cpp

@@ -8,14 +8,19 @@ int CommandSearch::handle(QStringList arguments)
 	QCommandLineParser parser;
 	parser.addOptions({
 		{{"r", "reverse"},
-		 "Print most-recent changed files first. This is short for adding \"sort:(mtime asc)\" to the query."},
+		 "Print most-recently changed files last. This is short for adding \"sort:(mtime asc)\" to the query."},
 	});
 
 	parser.addHelpOption();
 	parser.process(arguments);
 
-	QStringList files = parser.positionalArguments();
-	QString queryStrings = files.join(' ');
+	QStringList terms = parser.positionalArguments();
+	if(terms.length() == 0)
+	{
+		Logger::error() << "Please enter search terms" << Qt::endl;
+		return 1;
+	}
+	QString queryStrings = terms.join(' ');
 	LooqsQuery query = LooqsQuery::build(queryStrings, TokenType::FILTER_PATH_CONTAINS, false);
 	bool reverse = parser.isSet("reverse");
 	if(reverse)
@@ -26,11 +31,19 @@ int CommandSearch::handle(QStringList arguments)
 		query.addSortCondition(sc);
 	}
 
-	auto results = dbService->search(query);
-
-	for(SearchResult &result : results)
+	try
 	{
-		Logger::info() << result.fileData.absPath << Qt::endl;
+		auto results = dbService->search(query);
+
+		for(SearchResult &result : results)
+		{
+			Logger::info() << result.fileData.absPath << Qt::endl;
+		}
+	}
+	catch(LooqsGeneralException &e)
+	{
+		Logger::error() << "Exception:" << e.message << Qt::endl;
+		return 1;
 	}
 
 	return 0;

cli/commandupdate.cpp

@@ -4,6 +4,7 @@
 #include <QThreadPool>
 #include "commandupdate.h"
 #include "logger.h"
+#include "../shared/indexsyncer.h"
 
 int CommandUpdate::handle(QStringList arguments)
 {
@@ -12,18 +13,17 @@ int CommandUpdate::handle(QStringList arguments)
 		{{{"v", "verbose"}, "Print path of the files while updating them"},
 		 {{"n", "dry-run"}, "Only print which files would be updated, don't actually update them"},
 		 {"pattern", "Only consider to update files in the index matching the pattern, e. g. */.git/*.", "pattern"},
-		 {{"d", "delete"}, "If a file does not exist anymore, delete it"},
+		 {{"d", "delete"}, "If a file does not exist anymore, delete it from the index"},
 		 {{"c", "continue"},
 		  "Continue adding files, don't exit on first error. If this option is not given, looqs will exit asap, but "
 		  "it's possible that a few files will still be processed. "
 		  "Set -t 1 to avoid this behavior, but processing will be slower."},
-		 {{"a", "all"}, "On error, no files should be updated, even already processed ones"},
 		 {{"t", "threads"}, "Number of threads to use.", "threads"}
 
 		});
 
 	parser.addHelpOption();
-	parser.addPositionalArgument("update", "Checks files for changes and updates them", "update");
+	parser.addPositionalArgument("update", "Checks files for changes and updates the index", "update");
 
 	parser.process(arguments);
 	bool keepGoing = parser.isSet("continue");
@@ -32,84 +32,61 @@ int CommandUpdate::handle(QStringList arguments)
 	bool dryRun = parser.isSet("dry-run");
 	QString pattern = parser.value("pattern");
 
-	if(parser.isSet("all"))
-	{
-		throw LooqsGeneralException("To be implemented");
-	}
 	if(parser.isSet("threads"))
 	{
 		QString threadsCount = parser.value("threads");
 		QThreadPool::globalInstance()->setMaxThreadCount(threadsCount.toInt());
 	}
-	FileSaver saver(*this->dbService);
-	QVector<FileData> files;
-	int offset = 0;
-	int limit = 1000;
-	int processedRows = dbService->getFiles(files, pattern, offset, limit);
 
-	while(processedRows > 0)
+	bool hasErrors = false;
+	IndexSyncer *syncer = new IndexSyncer(*this->dbService);
+	syncer->setKeepGoing(keepGoing);
+	syncer->setVerbose(verbose);
+	syncer->setPattern(pattern);
+	syncer->setDryRun(dryRun);
+	syncer->setRemoveDeletedFromIndex(deleteMissing);
+
+	if(dryRun)
 	{
-		QVector<QString> filePathsToUpdate;
-		for(FileData &fileData : files)
-		{
-			QFileInfo fileInfo(fileData.absPath);
-			if(fileInfo.exists() && fileInfo.isFile())
-			{
-				if(fileInfo.lastModified().toSecsSinceEpoch() != fileData.mtime)
-				{
-					if(!dryRun)
-					{
-						filePathsToUpdate.append(fileData.absPath);
-					}
-					else
-					{
-						Logger::info() << "Would update" << fileData.absPath << Qt::endl;
-					}
-				}
-			}
-			else
-			{
-				if(deleteMissing)
-				{
-					if(!dryRun)
-					{
-						if(!this->dbService->deleteFile(fileData.absPath))
-						{
-							Logger::error()
-								<< "Error: Failed to delete" << fileData.absPath << "from databas" << Qt::endl;
-							if(!keepGoing)
-							{
-								return 1;
-							}
-						}
-						if(verbose)
-						{
-							Logger::info() << "Deleted" << fileData.absPath << Qt::endl;
-						}
-					}
-					else
-					{
-
-						Logger::info() << "Would delete" << fileData.absPath << Qt::endl;
-					}
-				}
-			}
-		}
-
-		int updatedFilesCount = saver.updateFiles(filePathsToUpdate, keepGoing, verbose);
-		int shouldHaveUpdatedCount = filePathsToUpdate.size();
-		if(updatedFilesCount != shouldHaveUpdatedCount)
-		{
-			if(!keepGoing)
-			{
-				Logger::error() << "Failed to update all files selected for updating. Updated" << updatedFilesCount
-								<< "out of" << shouldHaveUpdatedCount << "selected for upating" << Qt::endl;
-				return 1;
-			}
-		}
-		offset += limit;
-		processedRows = this->dbService->getFiles(files, pattern, offset, limit);
+		connect(syncer, &IndexSyncer::removedDryRun, this,
+				[](QString path) { Logger::info() << "Would delete" << path << Qt::endl; });
+		connect(syncer, &IndexSyncer::updatedDryRun, this,
+				[](QString path) { Logger::info() << "Would update" << path << Qt::endl; });
 	}
+	else
+	{
+		connect(syncer, &IndexSyncer::removed, this,
+				[](QString path) { Logger::info() << "Removed " << path << Qt::endl; });
+		/* TODO: updated not printed, handled be verbose in FileSaver, but this can be improved */
+	}
+	connect(syncer, &IndexSyncer::finished, this,
+			[&](unsigned int totalUpdated, unsigned int totalRemoved, unsigned int totalErrors)
+			{
+				Logger::info() << "Syncing finished" << Qt::endl;
 
+				if(!dryRun)
+				{
+					Logger::info() << "Total updated:" << totalUpdated << Qt::endl;
+					Logger::info() << "Total removed from index: " << totalRemoved << Qt::endl;
+					Logger::info() << "Total deleted:" << totalErrors << Qt::endl;
+				}
+
+				int retval = 0;
+				if(hasErrors && !keepGoing)
+				{
+					retval = 1;
+				}
+				emit finishedCmd(retval);
+			});
+	connect(syncer, &IndexSyncer::error, this,
+			[&](QString error)
+			{
+				Logger::error() << error << Qt::endl;
+				hasErrors = true;
+			});
+
+	this->autoFinish = false;
+	syncer->sync();
+	/* Actual return code is handled by finishedCmd signal */
 	return 0;
 }

cli/main.cpp

@@ -10,7 +10,6 @@
 #include <QSettings>
 #include <functional>
 #include <QTimer>
-
 #include <exception>
 #include "encodingdetector.h"
 #include "pdfprocessor.h"
@@ -21,6 +20,7 @@
 #include "commanddelete.h"
 #include "commandupdate.h"
 #include "commandsearch.h"
+#include "commandlist.h"
 #include "databasefactory.h"
 #include "logger.h"
 #include "sandboxedprocessor.h"
@@ -50,6 +50,10 @@ Command *commandFromName(QString name, SqliteDbService &dbService)
 	{
 		return new CommandSearch(dbService);
 	}
+	if(name == "list")
+	{
+		return new CommandList(dbService);
+	}
 
 	return nullptr;
 }

gui/gui.pro

@@ -23,7 +23,8 @@ DEFINES += QT_DEPRECATED_WARNINGS
 #DEFINES += QT_DISABLE_DEPRECATED_BEFORE=0x060000    # disables all the APIs deprecated before Qt 6.0.0
 
 SOURCES += \
-    ipcclient.cpp \
+    ipcpreviewclient.cpp \
+    ipcpreviewworker.cpp \
     ipcserver.cpp \
         main.cpp \
         mainwindow.cpp \
@@ -35,11 +36,13 @@ SOURCES += \
     previewresult.cpp \
     previewresultpdf.cpp \
     previewresultplaintext.cpp \
-    previewworker.cpp
+    renderconfig.cpp \
+    rendertarget.cpp
 
 HEADERS += \
     ipc.h \
-    ipcclient.h \
+    ipcpreviewclient.h \
+    ipcpreviewworker.h \
     ipcserver.h \
         mainwindow.h \
     clicklabel.h \
@@ -50,8 +53,8 @@ HEADERS += \
     previewresult.h \
     previewresultpdf.h \
     previewresultplaintext.h \
-    previewworker.h \
-    renderconfig.h
+    renderconfig.h \
+    rendertarget.h
 
 FORMS += \
         mainwindow.ui
@@ -74,3 +77,5 @@ else:win32-g++:CONFIG(debug, debug|release): PRE_TARGETDEPS += $$OUT_PWD/../shar
 else:win32:!win32-g++:CONFIG(release, debug|release): PRE_TARGETDEPS += $$OUT_PWD/../shared/release/shared.lib
 else:win32:!win32-g++:CONFIG(debug, debug|release): PRE_TARGETDEPS += $$OUT_PWD/../shared/debug/shared.lib
 else:unix: PRE_TARGETDEPS += $$OUT_PWD/../shared/libshared.a
+
+RESOURCES = ../icon.svg

gui/ipc.h

@@ -3,8 +3,13 @@
 
 enum IPCCommand
 {
-	DocOpen,
-	FileOpen,
-	AddFile,
+	GeneratePreviews = 23,
+	StopGeneratePreviews,
 };
+
+enum IPCReply
+{
+	FinishedGeneratePreviews,
+};
+
 #endif // IPC_H

gui/ipcclient.cpp

@@ -1,27 +0,0 @@
-#include <QDataStream>
-#include "ipcclient.h"
-
-IPCClient::IPCClient(QString socketPath)
-{
-	this->socketPath = socketPath;
-}
-
-bool IPCClient::sendCommand(IPCCommand command, QStringList args)
-{
-	bool result = false;
-	QLocalSocket socket;
-	socket.connectToServer(socketPath);
-	if(socket.isOpen() && socket.isWritable())
-	{
-		QDataStream stream(&socket);
-		stream << command;
-		stream << args;
-		socket.flush();
-		result = true;
-	}
-	else
-	{
-		qDebug() << "Not connected to IPC server";
-	}
-	return result;
-}

gui/ipcclient.h

@@ -1,18 +0,0 @@
-#ifndef IPCCLIENT_H
-#define IPCCLIENT_H
-#include <QLocalSocket>
-#include <QString>
-#include <QStringList>
-#include "ipc.h"
-
-class IPCClient
-{
-  private:
-	QString socketPath;
-
-  public:
-	IPCClient(QString socketPath);
-	bool sendCommand(IPCCommand command, QStringList args);
-};
-
-#endif // IPCCLIENT_H

gui/ipcpreviewclient.cpp

@@ -0,0 +1,130 @@
+#include <QLocalSocket>
+#include <QApplication>
+#include "ipc.h"
+#include "ipcpreviewclient.h"
+#include "previewresultpdf.h"
+#include "previewresultplaintext.h"
+
+bool IPCPreviewClient::connect()
+{
+	if(socket->state() == QLocalSocket::ConnectedState)
+	{
+		socket->disconnectFromServer();
+		if(socket->state() == QLocalSocket::ConnectedState)
+		{
+			socket->waitForDisconnected(100);
+		}
+	}
+	socket->connectToServer(socketPath);
+	socket->waitForConnected(100);
+	return socket->state() == QLocalSocket::ConnectedState;
+}
+
+QSharedPointer<PreviewResult> IPCPreviewClient::deserialize(QByteArray &array)
+{
+	QDataStream stream{&array, QIODevice::ReadOnly};
+
+	PreviewResultType type;
+	stream >> type;
+	if(type == PreviewResultType::PDF)
+	{
+		return PreviewResultPdf::deserialize(array);
+	}
+	if(type == PreviewResultType::PlainText)
+	{
+		return PreviewResultPlainText::deserialize(array);
+	}
+	return QSharedPointer<PreviewResult>(nullptr);
+}
+
+IPCPreviewClient::IPCPreviewClient()
+{
+	this->socket = new QLocalSocket(this);
+}
+
+void IPCPreviewClient::setSocketPath(QString socketPath)
+{
+	this->socketPath = socketPath;
+}
+
+void IPCPreviewClient::startGeneration(RenderConfig config, const QVector<RenderTarget> &targets)
+{
+	this->start(config, targets);
+}
+
+void IPCPreviewClient::start(RenderConfig config, const QVector<RenderTarget> &targets)
+{
+	if(targets.count() == 0)
+	{
+		return;
+	}
+
+	if(!connect() || !socket->isOpen())
+	{
+		emit error("Could not connect to IPC worker");
+		return;
+	}
+
+	if(socket->isOpen() && socket->isWritable())
+	{
+		QDataStream stream(socket);
+		stream << GeneratePreviews;
+		stream << config;
+		stream << targets;
+		socket->flush();
+
+		int numTarget = 0;
+		if(socket->isOpen() && socket->isReadable() && socket->state() == QLocalSocket::ConnectedState)
+		{
+			do
+			{
+				socket->waitForReadyRead(100);
+				stream.startTransaction();
+				stream >> numTarget;
+			} while(!stream.commitTransaction() && socket->state() == QLocalSocket::ConnectedState);
+			if(numTarget != targets.count())
+			{
+				emit error("IPC Error: Server reports less targets than it should");
+				return;
+			}
+		}
+		else
+		{
+			emit error("Error while trying to process previews: " + socket->errorString());
+			return;
+		}
+
+		int processed = 0;
+		++this->currentPreviewGeneration;
+		while(socket->isOpen() && socket->isReadable() && processed < targets.count())
+		{
+			QByteArray array;
+			do
+			{
+				socket->waitForReadyRead(100);
+				stream.startTransaction();
+				stream >> array;
+			} while(!stream.commitTransaction() && socket->state() == QLocalSocket::ConnectedState);
+			emit previewReceived(deserialize(array), this->currentPreviewGeneration);
+			++processed;
+		}
+		if(processed != targets.count())
+		{
+			emit error("IPC worker didn't send enough previews. This is a bug, please report");
+		}
+	}
+	socket->disconnectFromServer();
+	emit finished();
+}
+
+void IPCPreviewClient::stopGeneration()
+{
+	if(!connect() || !socket->isOpen())
+	{
+		emit error("Could not connect to IPC worker");
+		return;
+	}
+	QDataStream stream(socket);
+	stream << StopGeneratePreviews;
+	socket->flush();
+}

gui/ipcpreviewclient.h

@@ -0,0 +1,37 @@
+#ifndef IPCPREVIEWCLIENT_H
+#define IPCPREVIEWCLIENT_H
+#include <QObject>
+#include <QLocalSocket>
+#include "previewresult.h"
+#include "renderconfig.h"
+#include "rendertarget.h"
+
+class IPCPreviewClient : public QObject
+{
+	Q_OBJECT
+  private:
+	unsigned int currentPreviewGeneration = 1;
+	QLocalSocket *socket;
+	QString socketPath;
+
+	bool connect();
+	QSharedPointer<PreviewResult> deserialize(QByteArray &array);
+
+  public:
+	IPCPreviewClient();
+	~IPCPreviewClient()
+	{
+		delete socket;
+	}
+	void setSocketPath(QString socketPath);
+  public slots:
+	void start(RenderConfig config, const QVector<RenderTarget> &targets);
+	void startGeneration(RenderConfig config, const QVector<RenderTarget> &targets);
+	void stopGeneration();
+  signals:
+	void previewReceived(QSharedPointer<PreviewResult> previewResult, unsigned int currentPreviewGeneration);
+	void finished();
+	void error(QString);
+};
+
+#endif // IPCPREVIEWCLIENT_H

gui/ipcpreviewworker.cpp

@@ -0,0 +1,24 @@
+#include <QtConcurrent>
+#include "ipcpreviewworker.h"
+#include "previewgeneratormapfunctor.h"
+IPCPreviewWorker::IPCPreviewWorker()
+{
+	this->connect(&previewWorkerWatcher, &QFutureWatcher<QByteArray>::resultReadyAt, this,
+				  [this](int index) { emit previewGenerated(previewWorkerWatcher.resultAt(index)); });
+	connect(&previewWorkerWatcher, &QFutureWatcher<QByteArray>::finished, this, [this] { emit finished(); });
+}
+
+void IPCPreviewWorker::start(RenderConfig config, const QVector<RenderTarget> &targets, QLocalSocket *peer)
+{
+	stop();
+	auto mapFunctor = PreviewGeneratorMapFunctor();
+	mapFunctor.setRenderConfig(config);
+
+	previewWorkerWatcher.setFuture(QtConcurrent::mapped(targets, mapFunctor));
+}
+
+void IPCPreviewWorker::stop()
+{
+	previewWorkerWatcher.cancel();
+	previewWorkerWatcher.waitForFinished();
+}

gui/ipcpreviewworker.h

@@ -0,0 +1,24 @@
+#ifndef IPCPREVIEWWORKER_H
+#define IPCPREVIEWWORKER_H
+#include <QLocalSocket>
+#include <QFutureWatcher>
+#include "renderconfig.h"
+#include "rendertarget.h"
+#include "previewgenerator.h"
+
+class IPCPreviewWorker : public QObject
+{
+	Q_OBJECT
+  private:
+	QFutureWatcher<QByteArray> previewWorkerWatcher;
+
+  public:
+	IPCPreviewWorker();
+	void start(RenderConfig config, const QVector<RenderTarget> &targets, QLocalSocket *peer);
+	void stop();
+  signals:
+	void previewGenerated(QByteArray);
+	void finished();
+};
+
+#endif // IPCPREVIEWWORKER_H

gui/ipcserver.cpp

@@ -9,13 +9,17 @@
 #include "common.h"
 #include "databasefactory.h"
 #include "../shared/logger.h"
+#include "renderconfig.h"
+#include "rendertarget.h"
+#include "ipcpreviewworker.h"
 
 IpcServer::IpcServer()
 {
-	this->dbFactory = QSharedPointer<DatabaseFactory>(new DatabaseFactory(Common::databasePath()));
-	this->dbService = QSharedPointer<SqliteDbService>(new SqliteDbService(*this->dbFactory.get()));
-	this->fileSaver = QSharedPointer<FileSaver>(new FileSaver(*this->dbService.get()));
+	/* Only 1, we are doing work for the GUI, not a service for general availability */
+	this->spawningServer.setMaxPendingConnections(1);
 	connect(&this->spawningServer, &QLocalServer::newConnection, this, &IpcServer::spawnerNewConnection);
+	connect(&this->previewWorker, &IPCPreviewWorker::previewGenerated, this, &IpcServer::handlePreviewGenerated);
+	connect(&this->previewWorker, &IPCPreviewWorker::finished, this, [this] { this->currentSocket->flush(); });
 }
 
 bool IpcServer::startSpawner(QString socketPath)
@@ -24,85 +28,46 @@ bool IpcServer::startSpawner(QString socketPath)
 	return this->spawningServer.listen(socketPath);
 }
 
-bool IpcServer::docOpen(QString path, int pagenum)
-{
-
-	QSettings settings;
-	QString command = settings.value("pdfviewer").toString();
-	if(path.endsWith(".pdf") && command != "" && command.contains("%p") && command.contains("%f"))
-	{
-		QStringList splitted = command.split(" ");
-		if(splitted.size() > 1)
-		{
-			QString cmd = splitted[0];
-			QStringList args = splitted.mid(1);
-			args.replaceInStrings("%f", path);
-			args.replaceInStrings("%p", QString::number(pagenum));
-
-			QProcess::startDetached(cmd, args);
-		}
-	}
-	else
-	{
-		QDesktopServices::openUrl(QUrl::fromLocalFile(path));
-	}
-	return true;
-}
-
-bool IpcServer::fileOpen(QString path)
-{
-	return QDesktopServices::openUrl(QUrl::fromLocalFile(path));
-}
-
-SaveFileResult IpcServer::addFile(QString file)
-{
-	try
-	{
-		return this->fileSaver->addFile(file);
-	}
-	catch(std::exception &e)
-	{
-		Logger::error() << e.what() << Qt::endl;
-		return PROCESSFAIL;
-	}
-}
-
 void IpcServer::spawnerNewConnection()
 {
-	QScopedPointer<QLocalSocket> socket{this->spawningServer.nextPendingConnection()};
-	if(!socket.isNull())
+	QLocalSocket *socket = this->spawningServer.nextPendingConnection();
+	connect(socket, &QLocalSocket::disconnected, socket, &QLocalSocket::deleteLater);
+	this->currentSocket = socket;
+	if(socket != nullptr)
 	{
 		if(!socket->waitForReadyRead())
 		{
 			return;
 		}
-		QDataStream stream(socket.get());
+		QDataStream stream(socket);
 		IPCCommand command;
-		QStringList args;
 		stream >> command;
-		stream >> args;
-		if(args.size() < 1)
+		if(command == GeneratePreviews)
 		{
-			stream << "invalid";
-			return;
-		}
-		if(command == DocOpen)
-		{
-			if(args.size() < 2)
+			RenderConfig renderConfig;
+			QVector<RenderTarget> targets;
+			do
 			{
-				stream << "invalid";
-				return;
-			}
-			docOpen(args[0], args[1].toInt());
+				/* TODO: this is not entirely robust */
+				socket->waitForReadyRead(100);
+				stream.startTransaction();
+				stream >> renderConfig >> targets;
+			} while(!stream.commitTransaction() && socket->state() == QLocalSocket::ConnectedState);
+
+			stream << targets.count();
+			socket->flush();
+			previewWorker.start(renderConfig, targets, socket);
 		}
-		if(command == FileOpen)
+		if(command == StopGeneratePreviews)
 		{
-			if(args.size() < 1)
-			{
-				stream << "invalid";
-				return;
-			}
-			fileOpen(args[0]);
+			previewWorker.stop();
 		}
 	}
 }
+
+void IpcServer::handlePreviewGenerated(QByteArray ba)
+{
+	QDataStream stream{this->currentSocket};
+	stream << ba;
+	this->currentSocket->flush();
+}

gui/ipcserver.h

@@ -4,19 +4,19 @@
 #include <QLocalServer>
 #include "ipc.h"
 #include "filesaver.h"
+#include "ipcpreviewworker.h"
+
 class IpcServer : public QObject
 {
 	Q_OBJECT
   private:
-	QSharedPointer<DatabaseFactory> dbFactory;
-	QSharedPointer<SqliteDbService> dbService;
-	QSharedPointer<FileSaver> fileSaver;
+	IPCPreviewWorker previewWorker;
 	QLocalServer spawningServer;
-	bool docOpen(QString path, int pagenum);
-	bool fileOpen(QString path);
+	QLocalSocket *currentSocket = nullptr;
 	SaveFileResult addFile(QString file);
   private slots:
 	void spawnerNewConnection();
+	void handlePreviewGenerated(QByteArray ba);
 
   public:
 	IpcServer();

gui/main.cpp

@@ -5,6 +5,7 @@
 #include <QProcess>
 #include <QDir>
 #include <QCommandLineParser>
+#include <QFileInfo>
 
 #include "mainwindow.h"
 #include "searchresult.h"
@@ -14,63 +15,27 @@
 #include "../submodules/exile.h/exile.h"
 #include "ipcserver.h"
 
-void enableSandbox(QString socketPath)
+void enableIpcSandbox()
 {
-	struct exile_policy *policy = exile_init_policy();
+	struct exile_policy *policy = exile_create_policy();
 	if(policy == NULL)
 	{
 		qCritical() << "Failed to init policy for sandbox";
 		exit(EXIT_FAILURE);
 	}
-	QDir dir;
-	dir.mkpath(QStandardPaths::writableLocation(QStandardPaths::AppLocalDataLocation));
-	dir.mkpath(QStandardPaths::writableLocation(QStandardPaths::CacheLocation));
+	policy->namespace_options = EXILE_UNSHARE_NETWORK | EXILE_UNSHARE_USER;
+	policy->no_new_privs = 1;
+	policy->drop_caps = 1;
+	policy->vow_promises =
+		exile_vows_from_str("thread cpath wpath rpath unix stdio prot_exec proc shm fsnotify ioctl error");
 
-	std::string appDataLocation = QStandardPaths::writableLocation(QStandardPaths::AppLocalDataLocation).toStdString();
-	std::string cacheDataLocation = QStandardPaths::writableLocation(QStandardPaths::CacheLocation).toStdString();
-	std::string configDataLocation = QStandardPaths::writableLocation(QStandardPaths::ConfigLocation).toStdString();
+	QString ipcSocketPath = Common::ipcSocketPath();
+	QFileInfo info{ipcSocketPath};
+	QString ipcSocketPathDir = info.absolutePath();
+	std::string stdIpcSocketPath = ipcSocketPathDir.toStdString();
 
-	std::string sockPath = socketPath.toStdString();
-	std::string dbPath = QFileInfo(Common::databasePath()).absolutePath().toStdString();
-	std::string mySelf = QFileInfo("/proc/self/exe").symLinkTarget().toStdString();
-	policy->namespace_options = EXILE_UNSHARE_USER;
-	if(exile_append_path_policies(policy, EXILE_FS_ALLOW_ALL_READ, "/") != 0)
-	{
-		qCritical() << "Failed to append a path to the path policy";
-		exit(EXIT_FAILURE);
-	}
-
-	if(exile_append_path_policies(policy, EXILE_FS_ALLOW_ALL_READ | EXILE_FS_ALLOW_ALL_WRITE,
-								  appDataLocation.c_str()) != 0)
-	{
-		qCritical() << "Failed to append appDataLocation path to the path policy";
-		exit(EXIT_FAILURE);
-	}
-	if(exile_append_path_policies(policy, EXILE_FS_ALLOW_ALL_READ | EXILE_FS_ALLOW_ALL_WRITE,
-								  cacheDataLocation.c_str()) != 0)
-	{
-		qCritical() << "Failed to append cacheDataLocation path to the path policy";
-		exit(EXIT_FAILURE);
-	}
-	if(exile_append_path_policies(policy,
-								  EXILE_FS_ALLOW_ALL_READ | EXILE_FS_ALLOW_REMOVE_FILE | EXILE_FS_ALLOW_ALL_WRITE,
-								  dbPath.c_str()) != 0)
-	{
-		qCritical() << "Failed to append dbPath path to the path policy";
-		exit(EXIT_FAILURE);
-	}
-	if(exile_append_path_policies(policy, EXILE_FS_ALLOW_ALL_READ | EXILE_FS_ALLOW_EXEC, mySelf.c_str(), "/lib64",
-								  "/lib") != 0)
-	{
-		qCritical() << "Failed to append mySelf path to the path policy";
-		exit(EXIT_FAILURE);
-	}
-	if(exile_append_path_policies(policy, EXILE_FS_ALLOW_ALL_READ | EXILE_FS_ALLOW_ALL_WRITE,
-								  configDataLocation.c_str()) != 0)
-	{
-		qCritical() << "Failed to append configDataLocation path to the path policy";
-		exit(EXIT_FAILURE);
-	}
+	exile_append_path_policies(policy, EXILE_FS_ALLOW_ALL_READ, "/");
+	exile_append_path_policies(policy, EXILE_FS_ALLOW_ALL_READ | EXILE_FS_ALLOW_ALL_WRITE, stdIpcSocketPath.c_str());
 	int ret = exile_enable_policy(policy);
 	if(ret != 0)
 	{
@@ -79,15 +44,17 @@ void enableSandbox(QString socketPath)
 	}
 	exile_free_policy(policy);
 }
+
 int main(int argc, char *argv[])
 {
-	QString socketPath = "/tmp/looqs-spawner";
+	QString socketPath = Common::ipcSocketPath();
 	if(argc > 1)
 	{
 		QString arg = argv[1];
 		if(arg == "ipc")
 		{
 			Common::setupAppInfo();
+			enableIpcSandbox();
 			QApplication a(argc, argv);
 
 			IpcServer *ipcserver = new IpcServer();
@@ -103,24 +70,38 @@ int main(int argc, char *argv[])
 		if(arg == "process")
 		{
 			Common::setupAppInfo();
-			QApplication a(argc, argv);
+			QCoreApplication a(argc, argv);
 
 			QStringList args = a.arguments();
-			if(args.length() < 1)
+			if(args.length() < 3)
 			{
 				qDebug() << "Filename is required";
 				return 1;
 			}
 
-			QString file = args.at(1);
+			QString file = args.at(2);
 			SandboxedProcessor processor(file);
 			return processor.process();
 		}
 	}
+	QString ipcSocketPath = Common::ipcSocketPath();
+	QFileInfo info{ipcSocketPath};
+	QString ipcSocketPathDir = info.absolutePath();
+
+	QDir dir;
+	if(!dir.mkpath(ipcSocketPathDir))
+	{
+		qCritical() << "Failed to create dir for ipc socket" << Qt::endl;
+		exit(EXIT_FAILURE);
+	}
+
 	QProcess process;
 	QStringList args;
 	args << "ipc";
-	if(!process.startDetached("/proc/self/exe", args))
+	process.setProcessChannelMode(QProcess::ForwardedChannels);
+
+	process.start("/proc/self/exe", args);
+	if(!process.waitForStarted(5000))
 	{
 		QString errorMsg = "Failed to start IPC server";
 		qDebug() << errorMsg;
@@ -139,15 +120,6 @@ int main(int argc, char *argv[])
 	try
 	{
 		Common::ensureConfigured();
-		if(!parser.isSet("no-sandbox"))
-		{
-			enableSandbox(socketPath);
-			qInfo() << "Sandbox: on";
-		}
-		else
-		{
-			qInfo() << "Sandbox: off";
-		}
 	}
 	catch(LooqsGeneralException &e)
 	{
@@ -155,17 +127,20 @@ int main(int argc, char *argv[])
 		QMessageBox::critical(nullptr, "Error", e.message);
 		return 1;
 	}
-	// Keep this post sandbox, afterwards does not work (suspect due to threads, but unconfirmed)
 	QApplication a(argc, argv);
+	a.setWindowIcon(QIcon(":/icon.svg"));
+	QObject::connect(&a, &QApplication::aboutToQuit, &process, &QProcess::kill);
 
 	qRegisterMetaType<QVector<SearchResult>>("QVector<SearchResult>");
 	qRegisterMetaType<QVector<PreviewResultPdf>>("QVector<PreviewResultPdf>");
 	qRegisterMetaType<PreviewResultPdf>("PreviewResultPdf");
 	qRegisterMetaType<FileScanResult>("FileScanResult");
-
-	IPCClient client{socketPath};
-	MainWindow w{0, client};
-	w.showMaximized();
-
-	return a.exec();
+	qRegisterMetaType<RenderConfig>("RenderConfig");
+	qRegisterMetaType<QVector<RenderTarget>>("QVector<RenderTarget>");
+	qRegisterMetaType<QSharedPointer<PreviewResult>>("QSharedPointer<PreviewResult>");
+	MainWindow *w = new MainWindow{0, socketPath};
+	w->showMaximized();
+	int ret = a.exec();
+	process.waitForFinished(1000);
+	return ret;
 }

gui/mainwindow.cpp

@@ -12,24 +12,55 @@
 #include <QComboBox>
 #include <QtConcurrent/QtConcurrent>
 #include <QMessageBox>
+#include <QFileDialog>
+#include <QScreen>
+#include <QProgressDialog>
 #include "mainwindow.h"
 #include "ui_mainwindow.h"
 #include "clicklabel.h"
 #include "../shared/sqlitesearch.h"
 #include "../shared/looqsgeneralexception.h"
 #include "../shared/common.h"
+#include "ipcpreviewclient.h"
+#include "previewgenerator.h"
 
-MainWindow::MainWindow(QWidget *parent, IPCClient &client) : QMainWindow(parent), ui(new Ui::MainWindow)
+MainWindow::MainWindow(QWidget *parent, QString socketPath)
+	: QMainWindow(parent), ui(new Ui::MainWindow), progressDialog(this)
 {
+	this->progressDialog.cancel(); // because constructing it shows it, quite weird
 	ui->setupUi(this);
 	setWindowTitle(QCoreApplication::applicationName());
-	this->ipcClient = &client;
+	this->ipcPreviewClient.moveToThread(&this->ipcClientThread);
+	this->ipcPreviewClient.setSocketPath(socketPath);
+
+	connect(&ipcPreviewClient, &IPCPreviewClient::previewReceived, this, &MainWindow::previewReceived,
+			Qt::QueuedConnection);
+	connect(&ipcPreviewClient, &IPCPreviewClient::finished, this,
+			[&]
+			{
+				this->ui->previewProcessBar->setValue(this->ui->previewProcessBar->maximum());
+				this->ui->spinPreviewPage->setEnabled(true);
+			});
+	connect(&ipcPreviewClient, &IPCPreviewClient::error, this,
+			[this](QString msg)
+			{
+				qCritical() << msg << Qt::endl;
+				QMessageBox::critical(this, "IPC error", msg);
+			});
+
+	connect(this, &MainWindow::startIpcPreviews, &ipcPreviewClient, &IPCPreviewClient::startGeneration,
+			Qt::QueuedConnection);
+	connect(this, &MainWindow::stopIpcPreviews, &ipcPreviewClient, &IPCPreviewClient::stopGeneration,
+			Qt::QueuedConnection);
+	this->ipcClientThread.start();
 	QSettings settings;
 
 	this->dbFactory = new DatabaseFactory(Common::databasePath());
 
 	db = this->dbFactory->forCurrentThread();
 	this->dbService = new SqliteDbService(*this->dbFactory);
+	this->indexSyncer = new IndexSyncer(*this->dbService);
+	this->indexSyncer->moveToThread(&this->syncerThread);
 
 	indexer = new Indexer(*(this->dbService));
 	indexer->setParent(this);
@@ -45,6 +76,12 @@ MainWindow::MainWindow(QWidget *parent, IPCClient &client) : QMainWindow(parent)
 
 	QStringList indexPaths = settings.value("indexPaths").toStringList();
 	ui->lstPaths->addItems(indexPaths);
+
+	QString ignorePatterns = settings.value("ignorePatterns").toString();
+	ui->txtIgnorePatterns->setText(ignorePatterns);
+
+	ui->spinPreviewPage->setValue(1);
+	ui->spinPreviewPage->setMinimum(1);
 }
 
 void MainWindow::addPathToIndex()
@@ -64,6 +101,7 @@ void MainWindow::addPathToIndex()
 	this->ui->lstPaths->addItem(path);
 	this->ui->txtPathScanAdd->clear();
 }
+
 void MainWindow::connectSignals()
 {
 	connect(ui->txtSearch, &QLineEdit::returnPressed, this, &MainWindow::lineEditReturnPressed);
@@ -81,16 +119,6 @@ void MainWindow::connectSignals()
 					handleSearchError(e.message);
 				}
 			});
-	connect(&previewWorkerWatcher, &QFutureWatcher<QSharedPointer<PreviewResult>>::resultReadyAt, this,
-			[&](int index) { previewReceived(previewWorkerWatcher.resultAt(index)); });
-	connect(&previewWorkerWatcher, &QFutureWatcher<QSharedPointer<PreviewResult>>::progressValueChanged,
-			ui->previewProcessBar, &QProgressBar::setValue);
-	connect(&previewWorkerWatcher, &QFutureWatcher<QSharedPointer<PreviewResult>>::started, this,
-			[&] { ui->indexerTab->setEnabled(false); });
-
-	connect(&previewWorkerWatcher, &QFutureWatcher<QSharedPointer<PreviewResult>>::finished, this,
-			[&] { ui->indexerTab->setEnabled(true); });
-
 	connect(ui->treeResultsList, &QTreeWidget::itemActivated, this, &MainWindow::treeSearchItemActivated);
 	connect(ui->treeResultsList, &QTreeWidget::customContextMenuRequested, this,
 			&MainWindow::showSearchResultsContextMenu);
@@ -129,8 +157,85 @@ void MainWindow::connectSignals()
 			{ ui->btnDeletePath->setEnabled(this->ui->lstPaths->selectedItems().count() > 0); });
 
 	connect(ui->btnDeletePath, &QPushButton::clicked, this, [&] { qDeleteAll(ui->lstPaths->selectedItems()); });
+	connect(ui->btnChoosePath, &QPushButton::clicked, this,
+			[&]
+			{
+				QFileDialog dialog(nullptr);
+				dialog.setFileMode(QFileDialog::Directory);
+				dialog.setOptions(QFileDialog::ShowDirsOnly);
+				if(dialog.exec())
+				{
+					auto paths = dialog.selectedFiles();
+					if(paths.size() == 1)
+					{
+						ui->lstPaths->addItem(paths[0]);
+					}
+				}
+			});
+	connect(
+		ui->menuAboutAction, &QAction::triggered, this,
+		[this](bool checked)
+		{
+			QString html = "<h2>looqs</h2>";
+			html += "Full-text search with previews for your files<br><br>";
+			html += "Copyright (c) 2018-2022: Albert Schwarzkopf<br><br>";
+			html += QString("Version: %1<br><br>").arg(Common::versionText());
+			html += "Contact: looqs at quitesimple dot org<br><br>";
+			html += "Website: <a href=\"https://quitesimple.org\">https://quitesimple.org</a><br><br>";
+			html +=
+				"Source code: <a "
+				"href=\"https://github.com/quitesimpleorg/looqs\">https://github.com/quitesimpleorg/looqs</a><br><br>";
+			html += "License: GPLv3<br><br><br>";
+			html += "looqs is open source and free of charge in the hope it will be useful. The author(s) do not "
+					"give any warranty. In the unlikely event of any damage, the author(s) cannot be held responsible. "
+					"You are using looqs at your own risk";
+
+			QMessageBox::about(this, "About looqs", html);
+		});
+	connect(ui->menuOpenConfigInTextEditorAction, &QAction::triggered, this,
+			[this](bool checked)
+			{
+				QSettings setting;
+				QDesktopServices::openUrl(setting.fileName());
+			});
+	connect(ui->menuAboutQtAction, &QAction::triggered, this,
+			[this](bool checked) { QMessageBox::aboutQt(this, "About Qt"); });
+	connect(ui->menuSyncIndexAction, &QAction::triggered, this, &MainWindow::startIndexSync);
+	connect(indexSyncer, &IndexSyncer::finished, this,
+			[&](unsigned int totalUpdated, unsigned int totalDeleted, unsigned int totalErrored)
+			{
+				this->progressDialog.cancel();
+
+				QMessageBox::information(
+					this, "Syncing finished",
+					QString("Syncing finished\n\nTotal updated: %1\nTotal deleted: %2\nTotal errors: %3\n")
+						.arg(QString::number(totalUpdated))
+						.arg(QString::number(totalDeleted))
+						.arg(QString::number(totalErrored)));
+			});
+	connect(this, &MainWindow::beginIndexSync, indexSyncer, &IndexSyncer::sync);
+	connect(&this->progressDialog, &QProgressDialog::canceled, indexSyncer, &IndexSyncer::cancel);
 }
 
+void MainWindow::startIndexSync()
+{
+	progressDialog.setWindowTitle("Syncing");
+	progressDialog.setLabelText("Syncing - this might take a moment, please wait");
+	progressDialog.setWindowModality(Qt::ApplicationModal);
+	progressDialog.setMinimum(0);
+	progressDialog.setMaximum(0);
+	progressDialog.setValue(0);
+	progressDialog.open();
+
+	indexSyncer->setKeepGoing(true);
+	indexSyncer->setVerbose(false);
+	indexSyncer->setDryRun(false);
+	indexSyncer->setRemoveDeletedFromIndex(true);
+
+	this->syncerThread.start();
+
+	emit beginIndexSync();
+}
 void MainWindow::spinPreviewPageValueChanged(int val)
 {
 	makePreviews(val);
@@ -162,9 +267,12 @@ void MainWindow::startIndexing()
 		pathSettingsValue.append(path);
 	}
 	this->indexer->setTargetPaths(paths);
+	QString ignorePatterns = ui->txtIgnorePatterns->text();
+	this->indexer->setIgnorePattern(ignorePatterns.split(";"));
 	this->indexer->beginIndexing();
 	QSettings settings;
 	settings.setValue("indexPaths", pathSettingsValue);
+	settings.setValue("ignorePatterns", ignorePatterns);
 	ui->btnStartIndexing->setText("Stop indexing");
 }
 
@@ -245,16 +353,21 @@ void MainWindow::tabChanged()
 	}
 }
 
-void MainWindow::previewReceived(QSharedPointer<PreviewResult> preview)
+void MainWindow::previewReceived(QSharedPointer<PreviewResult> preview, unsigned int previewGeneration)
 {
-	if(preview->hasPreview())
+	if(previewGeneration < this->currentPreviewGeneration)
+	{
+		return;
+	}
+	this->ui->previewProcessBar->setValue(this->ui->previewProcessBar->value() + 1);
+	if(!preview.isNull() && preview->hasPreview())
 	{
 		QString docPath = preview->getDocumentPath();
 		auto previewPage = preview->getPage();
 
 		ClickLabel *label = dynamic_cast<ClickLabel *>(preview->createPreviewWidget());
 		ui->scrollAreaWidgetContents->layout()->addWidget(label);
-		connect(label, &ClickLabel::leftClick, [this, docPath, previewPage]() { ipcDocOpen(docPath, previewPage); });
+		connect(label, &ClickLabel::leftClick, [this, docPath, previewPage]() { openDocument(docPath, previewPage); });
 		connect(label, &ClickLabel::rightClick,
 				[this, docPath, previewPage]()
 				{
@@ -276,6 +389,10 @@ void MainWindow::lineEditReturnPressed()
 		ui->lblSearchResults->setText("Invalid paranthesis");
 		return;
 	}
+	if(indexerTabActive())
+	{
+		ui->tabWidget->setCurrentIndex(0);
+	}
 	// TODO: validate q;
 	ui->treeResultsList->clear();
 	ui->lblSearchResults->setText("Searching...");
@@ -294,10 +411,20 @@ void MainWindow::lineEditReturnPressed()
 			if(addPathSearch)
 			{
 				LooqsQuery filesQuery = LooqsQuery::build(q, TokenType::FILTER_PATH_CONTAINS, false);
+				if(filesQuery.getLimit() == -1)
+				{
+					filesQuery.setLimit(1000);
+				}
+
 				results.append(searcher.search(filesQuery));
 			}
 			if(addContentSearch)
 			{
+				if(this->contentSearchQuery.getLimit() == -1)
+				{
+					this->contentSearchQuery.setLimit(1000);
+				}
+
 				results.append(searcher.search(this->contentSearchQuery));
 			}
 			return results;
@@ -327,9 +454,12 @@ void MainWindow::handleSearchResults(const QVector<SearchResult> &results)
 		bool exists = pathInfo.exists();
 		if(exists)
 		{
-			if(PreviewGenerator::get(pathInfo) != nullptr)
+			if(!pathInfo.suffix().contains("htm")) // hack until we can preview them properly...
 			{
-				this->previewableSearchResults.append(result);
+				if(PreviewGenerator::get(pathInfo) != nullptr)
+				{
+					this->previewableSearchResults.append(result);
+				}
 			}
 		}
 		else
@@ -339,12 +469,11 @@ void MainWindow::handleSearchResults(const QVector<SearchResult> &results)
 	}
 	ui->treeResultsList->resizeColumnToContents(0);
 	ui->treeResultsList->resizeColumnToContents(1);
+	ui->treeResultsList->resizeColumnToContents(2);
 	previewDirty = !this->previewableSearchResults.empty();
 
-	int numpages = ceil(static_cast<double>(this->previewableSearchResults.size()) / previewsPerPage);
-	ui->spinPreviewPage->setMinimum(1);
-	ui->spinPreviewPage->setMaximum(numpages);
 	ui->spinPreviewPage->setValue(1);
+
 	if(previewTabActive() && previewDirty)
 	{
 		makePreviews(1);
@@ -353,19 +482,17 @@ void MainWindow::handleSearchResults(const QVector<SearchResult> &results)
 	QString statusText = "Results: " + QString::number(results.size()) + " files";
 	if(hasDeleted)
 	{
-		statusText += " WARNING: Some files don't exist anymore. No preview available for those. Index out of sync";
+		statusText += " WARNING: Some files are inaccessible. No preview available for those. Index may be out of sync";
 	}
 	ui->lblSearchResults->setText(statusText);
 }
 
 void MainWindow::makePreviews(int page)
 {
-
-	this->previewWorkerWatcher.cancel();
-	this->previewWorkerWatcher.waitForFinished();
-
-	QCoreApplication::processEvents(); // Maybe not necessary anymore, depends on whether it's possible that a slot is
-									   // still to be fired.
+	if(this->previewableSearchResults.empty())
+	{
+		return;
+	}
 	qDeleteAll(ui->scrollAreaWidgetContents->children());
 
 	ui->scrollAreaWidgetContents->setLayout(new QHBoxLayout());
@@ -393,14 +520,43 @@ void MainWindow::makePreviews(int page)
 			}
 		}
 	}
-	PreviewWorker worker;
 	int end = previewsPerPage;
 	int begin = page * previewsPerPage - previewsPerPage;
-	this->previewWorkerWatcher.setFuture(worker.generatePreviews(this->previewableSearchResults.mid(begin, end),
-																 wordsToHighlight, scaleText.toInt() / 100.));
-	ui->previewProcessBar->setMaximum(this->previewWorkerWatcher.progressMaximum());
-	ui->previewProcessBar->setMinimum(this->previewWorkerWatcher.progressMinimum());
+	if(begin < 0)
+	{
+		// Should not happen actually
+		begin = 0;
+	}
+
+	RenderConfig renderConfig;
+	renderConfig.scaleX = QGuiApplication::primaryScreen()->physicalDotsPerInchX() * (scaleText.toInt() / 100.);
+	renderConfig.scaleY = QGuiApplication::primaryScreen()->physicalDotsPerInchY() * (scaleText.toInt() / 100.);
+	renderConfig.wordsToHighlight = wordsToHighlight;
+
+	QVector<RenderTarget> targets;
+	for(SearchResult &sr : this->previewableSearchResults)
+	{
+		RenderTarget renderTarget;
+		renderTarget.path = sr.fileData.absPath;
+
+		for(unsigned int pagenum : sr.pages)
+		{
+			renderTarget.page = (int)pagenum;
+			targets.append(renderTarget);
+		}
+	}
+	int numpages = ceil(static_cast<double>(targets.size()) / previewsPerPage);
+	ui->spinPreviewPage->setMaximum(numpages);
+	targets = targets.mid(begin, end);
+
+	ui->lblTotalPreviewPagesCount->setText(QString::number(numpages));
+	ui->previewProcessBar->setMaximum(targets.count());
+	ui->previewProcessBar->setMinimum(0);
+	ui->previewProcessBar->setValue(0);
 	ui->previewProcessBar->setVisible(this->previewableSearchResults.size() > 0);
+	++this->currentPreviewGeneration;
+	this->ui->spinPreviewPage->setEnabled(false);
+	emit startIpcPreviews(renderConfig, targets);
 }
 
 void MainWindow::handleSearchError(QString error)
@@ -414,27 +570,39 @@ void MainWindow::createSearchResutlMenu(QMenu &menu, const QFileInfo &fileInfo)
 				   [&fileInfo] { QGuiApplication::clipboard()->setText(fileInfo.fileName()); });
 	menu.addAction("Copy full path to clipboard",
 				   [&fileInfo] { QGuiApplication::clipboard()->setText(fileInfo.absoluteFilePath()); });
-	menu.addAction("Open containing folder", [this, &fileInfo] { this->ipcFileOpen(fileInfo.absolutePath()); });
+	menu.addAction("Open containing folder", [this, &fileInfo] { this->openFile(fileInfo.absolutePath()); });
 }
 
-void MainWindow::ipcDocOpen(QString path, int num)
+void MainWindow::openDocument(QString path, int num)
 {
-	QStringList args;
-	args << path;
-	args << QString::number(num);
-	this->ipcClient->sendCommand(DocOpen, args);
+	QSettings settings;
+	QString command = settings.value("pdfviewer").toString();
+	if(path.endsWith(".pdf") && command != "" && command.contains("%p") && command.contains("%f"))
+	{
+		QStringList splitted = command.split(" ");
+		if(splitted.size() > 1)
+		{
+			QString cmd = splitted[0];
+			QStringList args = splitted.mid(1);
+			args.replaceInStrings("%f", path);
+			args.replaceInStrings("%p", QString::number(num));
+			QProcess::startDetached(cmd, args);
+		}
+	}
+	else
+	{
+		QDesktopServices::openUrl(QUrl::fromLocalFile(path));
+	}
 }
 
-void MainWindow::ipcFileOpen(QString path)
+void MainWindow::openFile(QString path)
 {
-	QStringList args;
-	args << path;
-	this->ipcClient->sendCommand(FileOpen, args);
+	QDesktopServices::openUrl(QUrl::fromLocalFile(path));
 }
 
 void MainWindow::treeSearchItemActivated(QTreeWidgetItem *item, int i)
 {
-	ipcFileOpen(item->text(1));
+	openFile(item->text(1));
 }
 
 void MainWindow::showSearchResultsContextMenu(const QPoint &point)
@@ -452,5 +620,11 @@ void MainWindow::showSearchResultsContextMenu(const QPoint &point)
 
 MainWindow::~MainWindow()
 {
+	syncerThread.terminate();
+	ipcClientThread.terminate();
+	delete this->indexSyncer;
+	delete this->dbService;
+	delete this->dbFactory;
+	delete this->indexer;
 	delete ui;
 }

gui/mainwindow.h

@@ -9,9 +9,10 @@
 #include <QFutureWatcher>
 #include <QSqlDatabase>
 #include <QLocalSocket>
-#include "previewworker.h"
+#include <QProgressDialog>
 #include "../shared/looqsquery.h"
-#include "ipcclient.h"
+#include "../shared/indexsyncer.h"
+#include "ipcpreviewclient.h"
 #include "indexer.h"
 namespace Ui
 {
@@ -23,7 +24,7 @@ class MainWindow : public QMainWindow
 	Q_OBJECT
 
   public:
-	explicit MainWindow(QWidget *parent, IPCClient &client);
+	explicit MainWindow(QWidget *parent, QString socketPath);
 	~MainWindow();
   signals:
 	void beginSearch(const QString &query);
@@ -33,13 +34,17 @@ class MainWindow : public QMainWindow
 	DatabaseFactory *dbFactory;
 	SqliteDbService *dbService;
 	Ui::MainWindow *ui;
-	IPCClient *ipcClient;
+	IPCPreviewClient ipcPreviewClient;
+	QThread ipcClientThread;
+	QThread syncerThread;
+	IndexSyncer *indexSyncer;
+	QProgressDialog progressDialog;
+
 	Indexer *indexer;
 	QFileIconProvider iconProvider;
 	bool previewDirty;
 	QSqlDatabase db;
 	QFutureWatcher<QVector<SearchResult>> searchWatcher;
-	QFutureWatcher<QSharedPointer<PreviewResult>> previewWorkerWatcher;
 	void add(QString path, unsigned int page);
 	QVector<SearchResult> previewableSearchResults;
 	void connectSignals();
@@ -53,20 +58,26 @@ class MainWindow : public QMainWindow
 	LooqsQuery contentSearchQuery;
 	int previewsPerPage;
 	void createSearchResutlMenu(QMenu &menu, const QFileInfo &fileInfo);
-	void ipcDocOpen(QString path, int num);
-	void ipcFileOpen(QString path);
-
+	void openDocument(QString path, int num);
+	void openFile(QString path);
+	unsigned int currentPreviewGeneration = 1;
   private slots:
 	void lineEditReturnPressed();
 	void treeSearchItemActivated(QTreeWidgetItem *item, int i);
 	void showSearchResultsContextMenu(const QPoint &point);
 	void tabChanged();
-	void previewReceived(QSharedPointer<PreviewResult> preview);
+	void previewReceived(QSharedPointer<PreviewResult> preview, unsigned int previewGeneration);
 	void comboScaleChanged(int i);
 	void spinPreviewPageValueChanged(int val);
 	void startIndexing();
 	void finishIndexing();
 	void addPathToIndex();
+	void startIndexSync();
+
+  signals:
+	void startIpcPreviews(RenderConfig config, const QVector<RenderTarget> &targets);
+	void stopIpcPreviews();
+	void beginIndexSync();
 };
 
 #endif // MAINWINDOW_H

gui/mainwindow.ui

@@ -7,7 +7,7 @@
     <x>0</x>
     <y>0</y>
     <width>1221</width>
-    <height>674</height>
+    <height>709</height>
    </rect>
   </property>
   <property name="windowTitle">
@@ -27,7 +27,7 @@
        <enum>QTabWidget::South</enum>
       </property>
       <property name="currentIndex">
-       <number>2</number>
+       <number>0</number>
       </property>
       <widget class="QWidget" name="resultsTab">
        <attribute name="title">
@@ -155,6 +155,13 @@
             </property>
            </widget>
           </item>
+          <item>
+           <widget class="QLabel" name="lblTotalPreviewPagesCount">
+            <property name="text">
+             <string/>
+            </property>
+           </widget>
+          </item>
           <item>
            <spacer name="horizontalSpacer">
             <property name="orientation">
@@ -326,7 +333,7 @@
          </widget>
         </item>
         <item row="6" column="0">
-         <widget class="QLineEdit" name="lineEdit"/>
+         <widget class="QLineEdit" name="txtIgnorePatterns"/>
         </item>
         <item row="10" column="0">
          <widget class="QPushButton" name="btnStartIndexing">
@@ -349,13 +356,54 @@
     <item>
      <widget class="QProgressBar" name="previewProcessBar">
       <property name="value">
-       <number>24</number>
+       <number>0</number>
       </property>
      </widget>
     </item>
    </layout>
   </widget>
   <widget class="QStatusBar" name="statusBar"/>
+  <widget class="QMenuBar" name="menuBar">
+   <property name="geometry">
+    <rect>
+     <x>0</x>
+     <y>0</y>
+     <width>1221</width>
+     <height>35</height>
+    </rect>
+   </property>
+   <widget class="QMenu" name="menulooqs">
+    <property name="title">
+     <string>looqs</string>
+    </property>
+    <addaction name="menuOpenConfigInTextEditorAction"/>
+    <addaction name="menuSyncIndexAction"/>
+    <addaction name="menuAboutAction"/>
+    <addaction name="menuAboutQtAction"/>
+    <addaction name="separator"/>
+   </widget>
+   <addaction name="menulooqs"/>
+  </widget>
+  <action name="menuAboutAction">
+   <property name="text">
+    <string>About</string>
+   </property>
+  </action>
+  <action name="menuOpenConfigInTextEditorAction">
+   <property name="text">
+    <string>Open config in text editor</string>
+   </property>
+  </action>
+  <action name="menuAboutQtAction">
+   <property name="text">
+    <string>About Qt</string>
+   </property>
+  </action>
+  <action name="menuSyncIndexAction">
+   <property name="text">
+    <string>Sync index (remove deleted, update existing files)</string>
+   </property>
+  </action>
  </widget>
  <layoutdefault spacing="6" margin="11"/>
  <resources/>

gui/previewgenerator.cpp

@@ -1,3 +1,4 @@
+#include "../shared/common.h"
 #include "previewgenerator.h"
 #include "previewgeneratorpdf.h"
 #include "previewgeneratorplaintext.h"
@@ -11,5 +12,13 @@ static QMap<QString, PreviewGenerator *> generators{
 
 PreviewGenerator *PreviewGenerator::get(QFileInfo &info)
 {
-	return generators.value(info.suffix(), nullptr);
+	PreviewGenerator *result = generators.value(info.suffix(), nullptr);
+	if(result == nullptr)
+	{
+		if(Common::isTextFile(info))
+		{
+			return plainTextGenerator;
+		}
+	}
+	return result;
 }

gui/previewgenerator.h

@@ -9,7 +9,7 @@
 class PreviewGenerator
 {
   public:
-	virtual PreviewResult *generate(RenderConfig config, QString documentPath, unsigned int page) = 0;
+	virtual QSharedPointer<PreviewResult> generate(RenderConfig config, QString documentPath, unsigned int page) = 0;
 	virtual ~PreviewGenerator()
 	{
 	}

gui/previewgeneratormapfunctor.cpp

@@ -10,16 +10,15 @@ void PreviewGeneratorMapFunctor::setRenderConfig(RenderConfig config)
 	this->renderConfig = config;
 }
 
-QSharedPointer<PreviewResult> PreviewGeneratorMapFunctor::operator()(const QSharedPointer<PreviewResult> &renderResult)
+QByteArray PreviewGeneratorMapFunctor::operator()(const RenderTarget &renderTarget)
 {
-	QFileInfo info{renderResult->getDocumentPath()};
+	QFileInfo info{renderTarget.path};
 	PreviewGenerator *previewGenerator = PreviewGenerator::get(info);
 	if(previewGenerator == nullptr)
 	{
-		return QSharedPointer<PreviewResult>();
+		return QByteArray{};
 	}
-	auto preview =
-		previewGenerator->generate(this->renderConfig, renderResult->getDocumentPath(), renderResult->getPage());
+	auto preview = previewGenerator->generate(this->renderConfig, renderTarget.path, renderTarget.page);
 
-	return QSharedPointer<PreviewResult>(preview);
+	return preview->serialize();
 }

gui/previewgeneratormapfunctor.h

@@ -2,8 +2,9 @@
 #define PREVIEWGENERATORMAPFUNCTOR_H
 
 #include "renderconfig.h"
-#include "previewgenerator.h"
+#include "rendertarget.h"
 
+#include "previewgenerator.h"
 class PreviewGeneratorMapFunctor
 {
 
@@ -16,13 +17,13 @@ class PreviewGeneratorMapFunctor
 	RenderConfig renderConfig;
 
   public:
-	typedef QSharedPointer<PreviewResult> result_type;
+	typedef QByteArray result_type;
 
 	PreviewGeneratorMapFunctor();
 
 	void setRenderConfig(RenderConfig config);
 
-	QSharedPointer<PreviewResult> operator()(const QSharedPointer<PreviewResult> &renderResult);
+	QByteArray operator()(const RenderTarget &renderTarget);
 };
 
 #endif // PREVIEWGENERATORMAPFUNCTOR_H

gui/previewgeneratorpdf.cpp

@@ -1,6 +1,5 @@
 #include <QMutexLocker>
 #include <QPainter>
-
 #include "previewgeneratorpdf.h"
 
 static QMutex cacheMutex;
@@ -24,18 +23,18 @@ Poppler::Document *PreviewGeneratorPdf::document(QString path)
 	return result;
 }
 
-PreviewResult *PreviewGeneratorPdf::generate(RenderConfig config, QString documentPath, unsigned int page)
+QSharedPointer<PreviewResult> PreviewGeneratorPdf::generate(RenderConfig config, QString documentPath,
+															unsigned int page)
 {
 	PreviewResultPdf *result = new PreviewResultPdf(documentPath, page);
-
 	Poppler::Document *doc = document(documentPath);
 	if(doc == nullptr)
 	{
-		return result;
+		return QSharedPointer<PreviewResult>(result);
 	}
 	if(doc->isLocked())
 	{
-		return result;
+		return QSharedPointer<PreviewResult>(result);
 	}
 	int p = (int)page - 1;
 	if(p < 0)
@@ -55,5 +54,5 @@ PreviewResult *PreviewGeneratorPdf::generate(RenderConfig config, QString docume
 		}
 	}
 	result->previewImage = img;
-	return result;
+	return QSharedPointer<PreviewResult>(result);
 }

gui/previewgeneratorpdf.h

@@ -13,7 +13,7 @@ class PreviewGeneratorPdf : public PreviewGenerator
   public:
 	using PreviewGenerator::PreviewGenerator;
 
-	PreviewResult *generate(RenderConfig config, QString documentPath, unsigned int page);
+	QSharedPointer<PreviewResult> generate(RenderConfig config, QString documentPath, unsigned int page);
 
 	~PreviewGeneratorPdf()
 	{

gui/previewgeneratorplaintext.cpp

@@ -3,13 +3,14 @@
 #include "previewgeneratorplaintext.h"
 #include "previewresultplaintext.h"
 
-PreviewResult *PreviewGeneratorPlainText::generate(RenderConfig config, QString documentPath, unsigned int page)
+QSharedPointer<PreviewResult> PreviewGeneratorPlainText::generate(RenderConfig config, QString documentPath,
+																  unsigned int page)
 {
 	PreviewResultPlainText *result = new PreviewResultPlainText(documentPath, page);
 	QFile file(documentPath);
 	if(!file.open(QFile::ReadOnly | QFile::Text))
 	{
-		return result;
+		return QSharedPointer<PreviewResultPlainText>(result);
 	}
 	QTextStream in(&file);
 
@@ -22,12 +23,14 @@ PreviewResult *PreviewGeneratorPlainText::generate(RenderConfig config, QString
 	int lastWordPos = 0;
 	QHash<QString, int> countmap;
 
+	const unsigned int maxSnippets = 7;
+	unsigned int currentSnippets = 0;
 	for(QString &word : config.wordsToHighlight)
 	{
 
 		int lastPos = 0;
 		int index = content.indexOf(word, lastPos, Qt::CaseInsensitive);
-		while(index != -1)
+		while(index != -1 && currentSnippets < maxSnippets)
 		{
 			countmap[word] = countmap.value(word, 0) + 1;
 
@@ -51,6 +54,7 @@ PreviewResult *PreviewGeneratorPlainText::generate(RenderConfig config, QString
 			lastPos = index;
 
 			index = content.indexOf(word, lastPos + 1, Qt::CaseInsensitive);
+			++currentSnippets;
 		}
 		lastWordPos = lastPos;
 	}
@@ -74,8 +78,13 @@ PreviewResult *PreviewGeneratorPlainText::generate(RenderConfig config, QString
 	{
 		header += word + ": " + QString::number(countmap[word]) + " ";
 	}
+	if(currentSnippets == maxSnippets)
+	{
+		header += "(truncated)";
+	}
+
 	header += "<hr>";
 
-	result->setText(header + resulText.replace("\n", "<br>"));
-	return result;
+	result->setText(header + resulText.replace("\n", "<br>").mid(0, 1000));
+	return QSharedPointer<PreviewResultPlainText>(result);
 }

gui/previewgeneratorplaintext.h

@@ -6,7 +6,7 @@ class PreviewGeneratorPlainText : public PreviewGenerator
 {
   public:
 	using PreviewGenerator::PreviewGenerator;
-	PreviewResult *generate(RenderConfig config, QString documentPath, unsigned int page);
+	QSharedPointer<PreviewResult> generate(RenderConfig config, QString documentPath, unsigned int page);
 };
 
 #endif // PREVIEWGENERATORPLAINTEXT_H

gui/previewresult.cpp

@@ -1,5 +1,4 @@
 #include "previewresult.h"
-
 PreviewResult::PreviewResult()
 {
 }
@@ -33,3 +32,11 @@ unsigned int PreviewResult::getPage() const
 {
 	return this->page;
 }
+
+QByteArray PreviewResult::serialize() const
+{
+	QByteArray result;
+	QDataStream stream{&result, QIODevice::WriteOnly};
+	stream << 0 << this->documentPath << this->page;
+	return result;
+}

gui/previewresult.h

@@ -2,6 +2,12 @@
 #define PREVIEWRESULT_H
 #include "clicklabel.h"
 
+enum PreviewResultType
+{
+	PDF = 1,
+	PlainText
+};
+
 class PreviewResult
 {
   protected:
@@ -17,6 +23,7 @@ class PreviewResult
 	virtual bool hasPreview();
 	QString getDocumentPath() const;
 	unsigned int getPage() const;
+	virtual QByteArray serialize() const;
 };
 
 #endif // PREVIEWRESULT_H

gui/previewresultpdf.cpp

@@ -1,5 +1,4 @@
 #include "previewresultpdf.h"
-
 PreviewResultPdf::PreviewResultPdf(const PreviewResult &o)
 {
 	this->documentPath = o.getDocumentPath();
@@ -19,3 +18,27 @@ bool PreviewResultPdf::hasPreview()
 	bool result = !this->previewImage.isNull();
 	return result;
 }
+
+QByteArray PreviewResultPdf::serialize() const
+{
+	QByteArray result;
+	QDataStream stream{&result, QIODevice::WriteOnly};
+	PreviewResultType type = PreviewResultType::PDF;
+	stream << type << this->documentPath << this->page << this->previewImage;
+	return result;
+}
+
+QSharedPointer<PreviewResultPdf> PreviewResultPdf::deserialize(QByteArray &ba)
+{
+	PreviewResultPdf *result = new PreviewResultPdf();
+	PreviewResultType type;
+
+	QDataStream stream{&ba, QIODevice::ReadOnly};
+	stream >> type;
+	if(type != PreviewResultType::PDF)
+	{
+		throw std::runtime_error("Invalid byte array: Not a pdf preview");
+	}
+	stream >> result->documentPath >> result->page >> result->previewImage;
+	return QSharedPointer<PreviewResultPdf>(result);
+}

gui/previewresultpdf.h

@@ -12,6 +12,10 @@ class PreviewResultPdf : public PreviewResult
 
 	QWidget *createPreviewWidget() override;
 	bool hasPreview() override;
+
+	QByteArray serialize() const;
+
+	static QSharedPointer<PreviewResultPdf> deserialize(QByteArray &ba);
 };
 
 #endif // PREVIEWRESULTPDF_H

gui/previewresultplaintext.cpp

@@ -28,3 +28,27 @@ void PreviewResultPlainText::setText(QString text)
 {
 	this->text = text;
 }
+
+QByteArray PreviewResultPlainText::serialize() const
+{
+	QByteArray result;
+	QDataStream stream{&result, QIODevice::WriteOnly};
+	PreviewResultType type = PreviewResultType::PlainText;
+	stream << type << this->documentPath << this->page << this->text;
+	return result;
+}
+
+QSharedPointer<PreviewResultPlainText> PreviewResultPlainText::deserialize(QByteArray &ba)
+{
+	PreviewResultPlainText *result = new PreviewResultPlainText();
+	PreviewResultType type;
+
+	QDataStream stream{&ba, QIODevice::ReadOnly};
+	stream >> type;
+	if(type != PreviewResultType::PlainText)
+	{
+		throw std::runtime_error("Invalid byte array: Not a pdf preview");
+	}
+	stream >> result->documentPath >> result->page >> result->text;
+	return QSharedPointer<PreviewResultPlainText>(result);
+}

gui/previewresultplaintext.h

@@ -15,6 +15,9 @@ class PreviewResultPlainText : public PreviewResult
 	bool hasPreview() override;
 
 	void setText(QString text);
+
+	QByteArray serialize() const;
+	static QSharedPointer<PreviewResultPlainText> deserialize(QByteArray &ba);
 };
 
 #endif // PREVIEWRESULTPLAINTEXT_H

gui/previewworker.cpp

@@ -1,39 +0,0 @@
-#include <QApplication>
-#include <QScreen>
-#include <QScopedPointer>
-#include <QMutexLocker>
-#include <QtConcurrent/QtConcurrent>
-#include <QtConcurrent/QtConcurrentMap>
-#include <atomic>
-#include "previewworker.h"
-
-PreviewWorker::PreviewWorker()
-{
-}
-
-QFuture<QSharedPointer<PreviewResult>> PreviewWorker::generatePreviews(const QVector<SearchResult> paths,
-																	   QVector<QString> wordsToHighlight,
-																	   double scalefactor)
-{
-	QVector<QSharedPointer<PreviewResult>> previews;
-
-	for(const SearchResult &sr : paths)
-	{
-		for(unsigned int page : sr.pages)
-		{
-			QSharedPointer<PreviewResult> ptr =
-				QSharedPointer<PreviewResult>(new PreviewResult{sr.fileData.absPath, page});
-			previews.append(ptr);
-		}
-	}
-
-	RenderConfig renderConfig;
-	renderConfig.scaleX = QGuiApplication::primaryScreen()->physicalDotsPerInchX() * scalefactor;
-	renderConfig.scaleY = QGuiApplication::primaryScreen()->physicalDotsPerInchY() * scalefactor;
-	renderConfig.wordsToHighlight = wordsToHighlight;
-
-	auto mapFunctor = new PreviewGeneratorMapFunctor();
-	mapFunctor->setRenderConfig(renderConfig);
-
-	return QtConcurrent::mapped(previews, *mapFunctor);
-}

gui/previewworker.h

@@ -1,29 +0,0 @@
-#ifndef PREVIEWWORKER_H
-#define PREVIEWWORKER_H
-#include <QObject>
-#include <QImage>
-#include <QHash>
-#include <QThread>
-#include <QMutex>
-#include <QWaitCondition>
-#include <QMutex>
-#include <QFuture>
-#include "previewresultpdf.h"
-#include "searchresult.h"
-#include "previewgenerator.h"
-#include "previewworker.h"
-#include "previewgeneratorpdf.h"
-#include "previewgeneratormapfunctor.h"
-
-class PreviewWorker : public QObject
-{
-	Q_OBJECT
-  public:
-	PreviewWorker();
-	QSharedPointer<PreviewGenerator> createGenerator(QString path);
-
-	QFuture<QSharedPointer<PreviewResult>> generatePreviews(const QVector<SearchResult> paths,
-															QVector<QString> wordsToHighlight, double scalefactor);
-};
-
-#endif // PREVIEWWORKER_H

gui/renderconfig.cpp

@@ -0,0 +1,17 @@
+#include "renderconfig.h"
+
+QDataStream &operator<<(QDataStream &out, const RenderConfig &rc)
+{
+	out << rc.scaleX;
+	out << rc.scaleY;
+	out << rc.wordsToHighlight;
+	return out;
+}
+
+QDataStream &operator>>(QDataStream &in, RenderConfig &rc)
+{
+	in >> rc.scaleX;
+	in >> rc.scaleY;
+	in >> rc.wordsToHighlight;
+	return in;
+}

gui/renderconfig.h

@@ -1,12 +1,17 @@
 #ifndef RENDERCONFIG_H
 #define RENDERCONFIG_H
 #include <QVector>
+#include <QDataStream>
 
 struct RenderConfig
 {
 	double scaleX = 50 / 100.;
 	double scaleY = scaleX;
 	QVector<QString> wordsToHighlight;
+	friend QDataStream &operator<<(QDataStream &out, const RenderConfig &rc);
+	friend QDataStream &operator>>(QDataStream &in, RenderConfig &rc);
 };
 
+QDataStream &operator<<(QDataStream &out, const RenderConfig &rc);
+QDataStream &operator>>(QDataStream &in, RenderConfig &rc);
 #endif // RENDERCONFIG_H

gui/rendertarget.cpp

@@ -0,0 +1,14 @@
+#include <QDataStream>
+
+#include "rendertarget.h"
+QDataStream &operator<<(QDataStream &out, const RenderTarget &rc)
+{
+	out << rc.path << rc.page;
+	return out;
+}
+
+QDataStream &operator>>(QDataStream &in, RenderTarget &rc)
+{
+	in >> rc.path >> rc.page;
+	return in;
+}

gui/rendertarget.h

@@ -0,0 +1,15 @@
+#ifndef RENDERTARGET_H
+#define RENDERTARGET_H
+#include <QString>
+struct RenderTarget
+{
+  public:
+	QString path;
+	int page;
+	friend QDataStream &operator<<(QDataStream &out, const RenderTarget &rc);
+	friend QDataStream &operator>>(QDataStream &in, RenderTarget &rc);
+};
+
+QDataStream &operator<<(QDataStream &out, const RenderTarget &rc);
+QDataStream &operator>>(QDataStream &in, RenderTarget &rc);
+#endif // RENDERTARGET_H

shared/common.cpp

@@ -7,6 +7,7 @@
 #include <QSqlError>
 #include <QTextStream>
 #include <QDebug>
+#include <QMimeDatabase>
 #include "looqsgeneralexception.h"
 #include "common.h"
 #include "dbmigrator.h"
@@ -16,6 +17,8 @@
 #define SETTINGS_KEY_DBPATH "dbpath"
 #define SETTINGS_KEY_FIRSTRUN "firstrun"
 #define SETTINGS_KEY_IPCSOCKETPATH "ipcsocketpath"
+#define SETTINGS_KEY_PDFVIEWER "pdfviewer"
+#define SETTINGS_KEY_EXCLUDEDPATHS "excludedpaths"
 
 inline void initResources()
 {
@@ -38,6 +41,52 @@ bool Common::initSqliteDatabase(QString path)
 	return true;
 }
 
+QString Common::findInPath(QString needle)
+{
+	QStringList results;
+	QString pathVar = QProcessEnvironment::systemEnvironment().value("PATH", "/usr/bin/:/bin/:");
+	QStringList paths = pathVar.split(":");
+	for(const QString &path : paths)
+	{
+		// TODO: can pass ../ but so be it for now.
+
+		QFileInfo info{path + "/" + needle};
+		if(info.exists())
+		{
+			return info.absoluteFilePath();
+		}
+	}
+	return "";
+}
+
+void Common::setPdfViewer()
+{
+	QString value;
+
+	/* TODO: well, we should query this probably from xdg*/
+	QString okularPath = findInPath("okular");
+	QString evincePath = findInPath("evince");
+	QString qpdfviewPath = findInPath("qpdfview");
+
+	if(okularPath != "")
+	{
+		value = okularPath + " %f -p %p";
+	}
+	else if(evincePath != "")
+	{
+		value = evincePath + "-i %p %f";
+	}
+	else if(qpdfviewPath != "")
+	{
+		value = qpdfviewPath + "%f#%p";
+	}
+
+	QSettings settings;
+	if(value != "")
+	{
+		settings.setValue(SETTINGS_KEY_PDFVIEWER, value);
+	}
+}
 void Common::ensureConfigured()
 {
 	QSettings settings;
@@ -58,8 +107,10 @@ void Common::ensureConfigured()
 		{
 			throw LooqsGeneralException("Failed to initialize sqlite database");
 		}
+
 		settings.setValue(SETTINGS_KEY_FIRSTRUN, false);
 		settings.setValue(SETTINGS_KEY_DBPATH, dbpath);
+		setPdfViewer();
 	}
 	else
 	{
@@ -107,6 +158,53 @@ QString Common::databasePath()
 
 QString Common::ipcSocketPath()
 {
-	QSettings settings;
-	return settings.value(SETTINGS_KEY_IPCSOCKETPATH, "/tmp/looqs-spawner").toString();
+	return "/tmp/.looqs/looqs-ipc-socket";
+
+	/* May not a good idea to set it in the settings and probably nobody would ever bother to change it anyway */
+	// QSettings settings;
+	// return settings.value(SETTINGS_KEY_IPCSOCKETPATH, "/tmp/.looqs/looqs-ipc-socket").toString();
+}
+
+static QStringList excludedPaths = {"/proc", "/sys", "/dev", "/tmp", "/var/run", "/run"};
+
+QStringList Common::excludedPaths()
+{
+	static int ran = false;
+	if(!ran)
+	{
+		QSettings settings;
+		QStringList userExcludedPaths = settings.value(SETTINGS_KEY_EXCLUDEDPATHS).toStringList();
+		ran = true;
+		::excludedPaths.append(userExcludedPaths);
+	}
+	return ::excludedPaths;
+}
+
+bool Common::isTextFile(QFileInfo fileInfo)
+{
+	/* TODO: This is not sandboxed yet ... */
+	QMimeDatabase mimeDatabase;
+	QMimeType mimeType = mimeDatabase.mimeTypeForFile(fileInfo);
+	if(mimeType.name().startsWith("text/"))
+	{
+		return true;
+	}
+	else
+	{
+		for(QString &str : mimeType.allAncestors())
+		{
+			if(str.startsWith("text/"))
+			{
+				return true;
+			}
+		}
+	}
+	return false;
+}
+
+QString Common::versionText()
+{
+	QString commitid = GIT_COMMIT_ID;
+	QString tag = GIT_TAG;
+	return tag + " (" + commitid + ") built " + __DATE__ + " " + __TIME__;
 }

shared/common.h

@@ -1,13 +1,18 @@
 #ifndef COMMON_H
 #define COMMON_H
 #include <QCoreApplication>
-
+#include <QFileInfo>
 namespace Common
 {
 void setupAppInfo();
 QString databasePath();
 QString ipcSocketPath();
+void setPdfViewer();
+QString findInPath(QString needle);
 bool initSqliteDatabase(QString path);
 void ensureConfigured();
+QStringList excludedPaths();
+bool isTextFile(QFileInfo fileInfo);
+QString versionText();
 } // namespace Common
 #endif

shared/dirscanworker.cpp

@@ -7,7 +7,7 @@ DirScanWorker::DirScanWorker(ConcurrentQueue<QString> &queue, ConcurrentQueue<QS
 {
 	this->queue = &queue;
 	this->resultQueue = &resultQueue;
-	this->ignorePattern = ignorePattern;
+	this->wildcardMatcher.setPatterns(ignorePattern);
 	this->progressReportThreshold = progressReportThreshold;
 	this->stopToken = &stopToken;
 	setAutoDelete(false);
@@ -24,10 +24,19 @@ void DirScanWorker::run()
 		start new DirScanWorkers ourselves here... */
 	while(queue->dequeue(path))
 	{
-		QDirIterator iterator(path, ignorePattern, QDir::Files, QDirIterator::Subdirectories);
+		if(wildcardMatcher.match(path))
+		{
+			continue;
+		}
+		QDirIterator iterator(path, QStringList{}, QDir::Files, QDirIterator::Subdirectories);
 		while(iterator.hasNext())
 		{
-			this->results.append(iterator.next());
+			QString entry = iterator.next();
+			if(wildcardMatcher.match(entry))
+			{
+				continue;
+			}
+			this->results.append(entry);
 			++currentProgress;
 			if(currentProgress == progressReportThreshold)
 			{

shared/dirscanworker.h

@@ -4,6 +4,7 @@
 #include <QRunnable>
 #include <QDirIterator>
 #include "concurrentqueue.h"
+#include "wildcardmatcher.h"
 class DirScanWorker : public QObject, public QRunnable
 {
 	Q_OBJECT
@@ -12,7 +13,7 @@ class DirScanWorker : public QObject, public QRunnable
 	ConcurrentQueue<QString> *queue = nullptr;
 	ConcurrentQueue<QString> *resultQueue = nullptr;
 
-	QStringList ignorePattern;
+	WildcardMatcher wildcardMatcher;
 	QVector<QString> results;
 
 	std::atomic<bool> *stopToken;

shared/filesaver.cpp

@@ -23,6 +23,7 @@ SaveFileResult FileSaver::addFile(QString path)
 {
 	QFileInfo info(path);
 	QString absPath = info.absoluteFilePath();
+
 	auto mtime = info.lastModified().toSecsSinceEpoch();
 	if(this->dbService->fileExistsInDatabase(absPath, mtime))
 	{
@@ -84,7 +85,7 @@ int FileSaver::processFiles(const QVector<QString> paths, std::function<SaveFile
 										  }
 										  else if(result == OK)
 										  {
-											  Logger::info() << "Added" << path << Qt::endl;
+											  Logger::info() << "Saved" << path << Qt::endl;
 										  }
 									  }
 								  }
@@ -95,7 +96,7 @@ int FileSaver::processFiles(const QVector<QString> paths, std::function<SaveFile
 SaveFileResult FileSaver::saveFile(const QFileInfo &fileInfo)
 {
 	QVector<PageData> pageData;
-	QString absPath = fileInfo.absoluteFilePath();
+	QString canonicalPath = fileInfo.canonicalFilePath();
 
 	int status = -1;
 
@@ -106,9 +107,17 @@ SaveFileResult FileSaver::saveFile(const QFileInfo &fileInfo)
 
 	if(fileInfo.isFile())
 	{
+		for(QString &excludedPath : this->excludedPaths)
+		{
+			if(canonicalPath.startsWith(excludedPath))
+			{
+				return SKIPPED;
+			}
+		}
+
 		QProcess process;
 		QStringList args;
-		args << "process" << absPath;
+		args << "process" << canonicalPath;
 		process.setProcessChannelMode(QProcess::ForwardedErrorChannel);
 		process.start("/proc/self/exe", args);
 		process.waitForStarted();
@@ -132,7 +141,7 @@ SaveFileResult FileSaver::saveFile(const QFileInfo &fileInfo)
 		status = process.exitCode();
 		if(status != 0 && status != NOTHING_PROCESSED)
 		{
-			Logger::error() << "FileSaver::saveFile(): Error while processing" << absPath << ":"
+			Logger::error() << "FileSaver::saveFile(): Error while processing" << canonicalPath << ":"
 							<< "Exit code " << status << Qt::endl;
 
 			return PROCESSFAIL;
@@ -142,7 +151,7 @@ SaveFileResult FileSaver::saveFile(const QFileInfo &fileInfo)
 	// Could happen if a file corrupted for example
 	if(pageData.isEmpty() && status != NOTHING_PROCESSED)
 	{
-		Logger::error() << "Could not get any content for " << absPath << Qt::endl;
+		Logger::error() << "Could not get any content for " << canonicalPath << Qt::endl;
 	}
 
 	return this->dbService->saveFile(fileInfo, pageData);

shared/filesaver.h

@@ -5,11 +5,12 @@
 #include "pagedata.h"
 #include "filedata.h"
 #include "sqlitedbservice.h"
-
+#include "common.h"
 class FileSaver
 {
   private:
 	SqliteDbService *dbService;
+	QStringList excludedPaths = Common::excludedPaths();
 
   public:
 	FileSaver(SqliteDbService &dbService);
@@ -20,8 +21,6 @@ class FileSaver
 					 bool keepGoing, bool verbose);
 	int addFiles(const QVector<QString> paths, bool keepGoing, bool verbose);
 	int updateFiles(const QVector<QString> paths, bool keepGoing, bool verbose);
-
-	;
 };
 
 #endif // FILESAVER_H

shared/indexer.cpp

@@ -1,5 +1,6 @@
 #include "indexer.h"
 #include "logger.h"
+#include "wildcardmatcher.h"
 
 Indexer::Indexer(SqliteDbService &db)
 {
@@ -17,8 +18,14 @@ void Indexer::beginIndexing()
 	this->currentIndexResult.begin = QDateTime::currentDateTime();
 	QVector<QString> dirs;
 
+	WildcardMatcher wildcardMatcher;
+	wildcardMatcher.setPatterns(this->ignorePattern);
 	for(QString &path : this->pathsToScan)
 	{
+		if(wildcardMatcher.match(path))
+		{
+			continue;
+		}
 		QFileInfo info{path};
 		if(info.isDir())
 		{

shared/indexsyncer.cpp

@@ -0,0 +1,129 @@
+#include <QDateTime>
+#include "filesaver.h"
+#include "indexsyncer.h"
+
+IndexSyncer::IndexSyncer(SqliteDbService &dbService)
+{
+	this->dbService = &dbService;
+}
+
+void IndexSyncer::setDryRun(bool dryRun)
+{
+	this->dryRun = dryRun;
+}
+
+void IndexSyncer::setVerbose(bool verbose)
+{
+	this->verbose = verbose;
+}
+
+void IndexSyncer::setKeepGoing(bool keepGoing)
+{
+	this->keepGoing = keepGoing;
+}
+
+void IndexSyncer::setRemoveDeletedFromIndex(bool removeDeletedFromIndex)
+{
+	this->removeDeletedFromIndex = removeDeletedFromIndex;
+}
+
+void IndexSyncer::setPattern(QString pattern)
+{
+	this->pattern = pattern;
+}
+
+void IndexSyncer::sync()
+{
+	this->stopToken.store(false, std::memory_order_relaxed);
+	FileSaver saver(*this->dbService);
+	QVector<FileData> files;
+	int offset = 0;
+	int limit = 10000;
+	unsigned int processedRows = dbService->getFiles(files, pattern, offset, limit);
+
+	unsigned int totalUpdatesFilesCount = 0;
+	unsigned int totalDeletedFilesCount = 0;
+	unsigned int totalErroredFilesCount = 0;
+
+	while(processedRows > 0)
+	{
+		QVector<QString> filePathsToUpdate;
+		for(FileData &fileData : files)
+		{
+			if(processedRows % 100 == 0 && this->stopToken.load(std::memory_order_relaxed))
+			{
+				emit finished(totalUpdatesFilesCount, totalDeletedFilesCount, totalErroredFilesCount);
+				return;
+			}
+			QFileInfo fileInfo(fileData.absPath);
+			if(fileInfo.exists())
+			{
+				if(fileInfo.isFile())
+				{
+					if(fileInfo.lastModified().toSecsSinceEpoch() != fileData.mtime)
+					{
+						if(!dryRun)
+						{
+							filePathsToUpdate.append(fileData.absPath);
+						}
+						else
+						{
+							emit updatedDryRun(fileData.absPath);
+						}
+					}
+				}
+			}
+			else
+			{
+				if(this->removeDeletedFromIndex)
+				{
+					if(!dryRun)
+					{
+						if(!this->dbService->deleteFile(fileData.absPath))
+						{
+							emit error("Error: Failed to delete " + fileData.absPath + " from the index");
+							if(!this->keepGoing)
+							{
+								emit finished(totalUpdatesFilesCount, totalDeletedFilesCount, totalErroredFilesCount);
+								return;
+							}
+						}
+						emit removed(fileData.absPath);
+						++totalDeletedFilesCount;
+					}
+					else
+					{
+						emit removedDryRun(fileData.absPath);
+					}
+				}
+			}
+		}
+
+		unsigned int updatedFilesCount = saver.updateFiles(filePathsToUpdate, keepGoing, verbose);
+		unsigned int shouldHaveUpdatedCount = static_cast<unsigned int>(filePathsToUpdate.size());
+		if(updatedFilesCount != shouldHaveUpdatedCount)
+		{
+
+			totalErroredFilesCount += (shouldHaveUpdatedCount - updatedFilesCount);
+			if(!keepGoing)
+			{
+				QString errorMsg = QString("Failed to update all files selected for updating in this batch. Updated") +
+								   updatedFilesCount + "out of" + shouldHaveUpdatedCount + "selected for updating";
+				emit error(errorMsg);
+				emit finished(totalUpdatesFilesCount, totalDeletedFilesCount, totalErroredFilesCount);
+			}
+		}
+		offset += limit;
+		files.clear();
+		totalUpdatesFilesCount += updatedFilesCount;
+
+		processedRows = this->dbService->getFiles(files, pattern, offset, limit);
+	}
+
+	emit finished(totalUpdatesFilesCount, totalDeletedFilesCount, totalErroredFilesCount);
+}
+
+void IndexSyncer::cancel()
+{
+	this->stopToken.store(true, std::memory_order_seq_cst);
+}

shared/indexsyncer.h

@@ -0,0 +1,39 @@
+#ifndef INDEXSYNCER_H
+#define INDEXSYNCER_H
+#include "sqlitedbservice.h"
+
+class IndexSyncer : public QObject
+{
+	Q_OBJECT
+  private:
+	SqliteDbService *dbService = nullptr;
+	bool keepGoing = true;
+	bool removeDeletedFromIndex = true;
+	bool dryRun = false;
+	bool verbose = false;
+	QString pattern;
+
+	std::atomic<bool> stopToken{false};
+
+  public:
+	IndexSyncer(SqliteDbService &dbService);
+
+  public slots:
+	void sync();
+	void cancel();
+	void setDryRun(bool dryRun);
+	void setVerbose(bool verbose);
+	void setKeepGoing(bool keepGoing);
+	void setRemoveDeletedFromIndex(bool removeDeletedFromIndex);
+	void setPattern(QString pattern);
+
+  signals:
+	void error(QString error);
+	void removed(QString path);
+	void removedDryRun(QString path);
+	void updatedDryRun(QString path);
+	void updated(QString path);
+	void finished(unsigned int totalUpdated, unsigned int totalDeleted, unsigned int totalErrored);
+};
+
+#endif // INDEXSYNCER_H

shared/looqsquery.cpp

@@ -169,6 +169,10 @@ void LooqsQuery::addToken(Token t)
  * TODO: It's a bit ugly still*/
 LooqsQuery LooqsQuery::build(QString expression, TokenType loneWordsTokenType, bool mergeLoneWords)
 {
+	if(expression.isEmpty())
+	{
+		return LooqsQuery{};
+	}
 	if(!checkParanthesis(expression))
 	{
 		throw LooqsGeneralException("Invalid paranthesis");
@@ -177,7 +181,7 @@ LooqsQuery LooqsQuery::build(QString expression, TokenType loneWordsTokenType, b
 	QStringList loneWords;
 	LooqsQuery result;
 	QRegularExpression rx("((?<filtername>(\\.|\\w)+):(?<args>\\((?<innerargs>[^\\)]+)\\)|([\\w,])+)|(?<boolean>AND|OR)"
-						  "|(?<negation>!)|(?<bracket>\\(|\\))|(?<loneword>\\w+))");
+						  "|(?<negation>!)|(?<bracket>\\(|\\))|(?<loneword>[\"\\w]+))");
 	QRegularExpressionMatchIterator i = rx.globalMatch(expression);
 	auto previousWasBool = [&result] { return !result.tokens.empty() && ((result.tokens.last().type & BOOL) == BOOL); };
 	auto previousWas = [&result](TokenType t) { return !result.tokens.empty() && (result.tokens.last().type == t); };
@@ -216,19 +220,8 @@ LooqsQuery LooqsQuery::build(QString expression, TokenType loneWordsTokenType, b
 			{
 				throw LooqsGeneralException("Can't have two negations following each other");
 			}
-			if(!previousWasBool())
-			{
-				result.addToken(Token(BOOL_AND)); // Implicit and, our default operation
-			}
 			result.addToken(Token(NEGATION));
 		}
-		if(!result.tokens.isEmpty() && !previousWasBool() && !previousWas(NEGATION) && !previousWas(BRACKET_OPEN) &&
-		   bracket != ")")
-		{
-			// the current token isn't a negation, isn't a boolean. Thus, implicit AND is required
-			result.addToken(Token(BOOL_AND));
-		}
-
 		if(bracket != "")
 		{
 			if(bracket == "(")
@@ -255,7 +248,7 @@ LooqsQuery LooqsQuery::build(QString expression, TokenType loneWordsTokenType, b
 
 		if(filtername != "")
 		{
-			TokenType tokenType;
+			TokenType tokenType = WORD;
 			QString value = m.captured("innerargs");
 			if(value == "")
 			{
@@ -266,15 +259,15 @@ LooqsQuery LooqsQuery::build(QString expression, TokenType loneWordsTokenType, b
 				throw LooqsGeneralException("value cannot be empty for filters");
 			}
 
-			if(filtername == "path.contains")
+			if(filtername == "p" || filtername == "path.contains")
 			{
 				tokenType = FILTER_PATH_CONTAINS;
 			}
-			else if(filtername == "path.starts")
+			else if(filtername == "pb" || filtername == "path.starts")
 			{
 				tokenType = FILTER_PATH_STARTS;
 			}
-			else if(filtername == "path.ends")
+			else if(filtername == "pe" || filtername == "path.ends")
 			{
 				tokenType = FILTER_PATH_ENDS;
 			}
@@ -290,21 +283,21 @@ LooqsQuery LooqsQuery::build(QString expression, TokenType loneWordsTokenType, b
 			{
 				tokenType = FILTER_CONTENT_PAGE;
 			}
-			else if(filtername ==
-					"sort") // TODO: given this is not really a "filter", this feels slightly misplaced here
+			// TODO: given this is not really a "filter", this feels slightly misplaced here
+			else if(filtername == "sort")
 			{
 				if(!result.sortConditions.empty())
 				{
 					throw LooqsGeneralException("Two sort statements are illegal");
 				}
-				// TODO: hack, since we are not a "filter", we must remove a preceeding (implicit) boolean
-				if((result.tokens.last().type & BOOL) == BOOL)
-				{
-					result.tokens.pop_back();
-				}
 				result.sortConditions = createSortConditions(value);
 				continue;
 			}
+			else if(filtername == "limit")
+			{
+				result.limit = value.toInt();
+				continue;
+			}
 			else
 			{
 				throw LooqsGeneralException("Unknown filter provided!");
@@ -322,6 +315,26 @@ LooqsQuery LooqsQuery::build(QString expression, TokenType loneWordsTokenType, b
 		}
 	}
 
+	/* Add our default implicit AND boolean condition where appropriate */
+	QVector<Token> newTokens;
+
+	TokenType prevType = BOOL_AND;
+	int needsBoolean = FILTER_CONTENT | FILTER_PATH | NEGATION;
+	for(Token &t : result.tokens)
+	{
+		if(t.type == BRACKET_OPEN || t.type & needsBoolean)
+		{
+			if(!((prevType & BOOL) == BOOL) && !((prevType & NEGATION) == NEGATION) &&
+			   !((prevType & BRACKET_OPEN) == BRACKET_OPEN))
+			{
+				newTokens.append(Token(BOOL_AND));
+			}
+		}
+		prevType = t.type;
+		newTokens.append(t);
+	}
+	result.tokens = newTokens;
+
 	bool contentsearch = result.hasContentSearch();
 	bool sortsForContent = std::any_of(result.sortConditions.begin(), result.sortConditions.end(),
 									   [](SortCondition c) { return c.field == CONTENT_TEXT; });

shared/looqsquery.h

@@ -40,6 +40,7 @@ class LooqsQuery
 	/* Helper field to determine quertype as well as to quickly check what kind of filters etc.
 	 * are being used in this query*/
 	int tokensMask = 0;
+	int limit = -1;
 	QVector<Token> tokens;
 	QVector<SortCondition> sortConditions;
 	void addToken(Token t);
@@ -52,6 +53,14 @@ class LooqsQuery
 	{
 		return tokensMask;
 	}
+	int getLimit() const
+	{
+		return limit;
+	}
+	void setLimit(int limit)
+	{
+		this->limit = limit;
+	}
 	bool hasContentSearch();
 	bool hasPathSearch();
 

shared/paralleldirscanner.cpp

@@ -43,7 +43,6 @@ void ParallelDirScanner::handleWorkersProgress(unsigned int progress)
 
 void ParallelDirScanner::handleWorkersFinish()
 {
-	Logger::info() << "Worker finished";
 	// no mutexes required due to queued connection
 	++finishedWorkers;
 	if(this->stopToken.load(std::memory_order_seq_cst) || finishedWorkers == getThreadsNum())
@@ -51,6 +50,7 @@ void ParallelDirScanner::handleWorkersFinish()
 		running = false;
 		emit scanComplete();
 	}
+	delete QObject::sender();
 }
 
 unsigned int ParallelDirScanner::getThreadsNum() const
@@ -65,7 +65,6 @@ unsigned int ParallelDirScanner::getThreadsNum() const
 
 void ParallelDirScanner::scan()
 {
-	Logger::info() << "I am scanning";
 	this->stopToken.store(false, std::memory_order_relaxed);
 	this->finishedWorkers = 0;
 	this->processedPaths = 0;

shared/paralleldirscanner.h

@@ -41,6 +41,7 @@ class ParallelDirScanner : public QObject
 	void progress(int, int);
   public slots:
 	void cancel();
+  private slots:
 	void handleWorkersProgress(unsigned int progress);
 	void handleWorkersFinish();
 };

shared/sandboxedprocessor.cpp

@@ -9,6 +9,7 @@
 #include "odtprocessor.h"
 #include "odsprocessor.h"
 #include "../submodules/exile.h/exile.h"
+#include "common.h"
 #include "logger.h"
 
 static DefaultTextProcessor *defaultTextProcessor = new DefaultTextProcessor();
@@ -22,7 +23,7 @@ static QMap<QString, Processor *> processors{
 	{"py", defaultTextProcessor},	{"xml", nothingProcessor},	   {"html", tagStripperProcessor},
 	{"java", defaultTextProcessor}, {"js", defaultTextProcessor},  {"cpp", defaultTextProcessor},
 	{"c", defaultTextProcessor},	{"sql", defaultTextProcessor}, {"odt", odtProcessor},
-	{"ods", odsProcessor}};
+	{"ods", odsProcessor},			{"svg", nothingProcessor}};
 
 void SandboxedProcessor::enableSandbox(QString readablePath)
 {
@@ -34,9 +35,10 @@ void SandboxedProcessor::enableSandbox(QString readablePath)
 	}
 	policy->namespace_options = EXILE_UNSHARE_NETWORK | EXILE_UNSHARE_USER;
 
+	std::string readablePathLocation;
 	if(!readablePath.isEmpty())
 	{
-		std::string readablePathLocation = readablePath.toStdString();
+		readablePathLocation = readablePath.toStdString();
 		if(exile_append_path_policies(policy, EXILE_FS_ALLOW_ALL_READ, readablePathLocation.c_str()) != 0)
 		{
 			qCritical() << "Failed to add path policies";
@@ -50,7 +52,7 @@ void SandboxedProcessor::enableSandbox(QString readablePath)
 	int ret = exile_enable_policy(policy);
 	if(ret != 0)
 	{
-		qDebug() << "Failed to establish sandbox: " << ret;
+		qCritical() << "Failed to establish sandbox: " << ret;
 		exit(EXIT_FAILURE);
 	}
 	exile_free_policy(policy);
@@ -74,9 +76,16 @@ void SandboxedProcessor::printResults(const QVector<PageData> &pageData)
 int SandboxedProcessor::process()
 {
 	QFileInfo fileInfo(this->filePath);
-	Processor *processor = processors.value(fileInfo.suffix(), nothingProcessor);
-
-	if(processor == nothingProcessor)
+	Processor *processor = processors.value(fileInfo.suffix(), nullptr);
+	if(processor == nullptr)
+	{
+		/* TODO: Not sandboxed */
+		if(Common::isTextFile(fileInfo))
+		{
+			processor = defaultTextProcessor;
+		}
+	}
+	if(processor == nullptr || processor == nothingProcessor)
 	{
 		/* Nothing to do */
 		return NOTHING_PROCESSED;

shared/sandboxedprocessor.h

@@ -1,12 +1,14 @@
 #ifndef SANDBOXEDPROCESSOR_H
 #define SANDBOXEDPROCESSOR_H
 #include <QString>
+#include <QMimeDatabase>
 #include "pagedata.h"
 
 class SandboxedProcessor
 {
   private:
 	QString filePath;
+	QMimeDatabase mimeDatabase;
 
 	void enableSandbox(QString readablePath = "");
 	void printResults(const QVector<PageData> &pageData);

shared/shared.pro

@@ -16,12 +16,17 @@ CONFIG += c++17
 INCLUDEPATH += $$PWD/../sandbox/exile.h/
 INCLUDEPATH += /usr/include/poppler/qt5/ /usr/include/quazip5
 
+
+
 # The following define makes your compiler emit warnings if you use
 # any feature of Qt which has been marked as deprecated (the exact warnings
 # depend on your compiler). Please consult the documentation of the
 # deprecated API in order to know how to port your code away from it.
 DEFINES += QT_DEPRECATED_WARNINGS
 
+DEFINES += GIT_COMMIT_ID="\\\"$(shell git rev-parse --short HEAD)\\\""
+DEFINES += GIT_TAG="\\\"$(shell git describe --tags HEAD)\\\""
+
 # You can also make your code fail to compile if you use deprecated APIs.
 # In order to do so, uncomment the following line.
 # You can also select to disable deprecated APIs only up to a certain version of Qt.
@@ -37,6 +42,7 @@ SOURCES += sqlitesearch.cpp \
     filesaver.cpp \
     filescanworker.cpp \
     indexer.cpp \
+    indexsyncer.cpp \
     logger.cpp \
     looqsgeneralexception.cpp \
     common.cpp \
@@ -52,7 +58,8 @@ SOURCES += sqlitesearch.cpp \
     sqlitedbservice.cpp \
     tagstripperprocessor.cpp \
     utils.cpp \
-    ../submodules/exile.h/exile.c
+    ../submodules/exile.h/exile.c \
+    wildcardmatcher.cpp
 
 HEADERS += sqlitesearch.h \
     concurrentqueue.h \
@@ -65,6 +72,7 @@ HEADERS += sqlitesearch.h \
     filesaver.h \
     filescanworker.h \
     indexer.h \
+    indexsyncer.h \
     logger.h \
     looqsgeneralexception.h \
     looqsquery.h \
@@ -81,7 +89,8 @@ HEADERS += sqlitesearch.h \
     tagstripperprocessor.h \
     token.h \
     common.h \
-    utils.h
+    utils.h \
+    wildcardmatcher.h
 unix {
     target.path = /usr/lib
     INSTALLS += target

shared/sqlitedbservice.cpp

@@ -62,10 +62,11 @@ bool SqliteDbService::deleteFile(QString path)
 	}
 	return result;
 }
-int SqliteDbService::getFiles(QVector<FileData> &results, QString wildCardPattern, int offset, int limit)
+
+unsigned int SqliteDbService::getFiles(QVector<FileData> &results, QString wildCardPattern, int offset, int limit)
 {
 
-	int processedRows = 0;
+	unsigned int processedRows = 0;
 	// TODO: translate/convert wildCardPattern to SQL where instead of regex
 	QString sql = "SELECT path, mtime, size, filetype FROM file";
 

shared/sqlitedbservice.h

@@ -24,7 +24,7 @@ class SqliteDbService
   public:
 	SqliteDbService(DatabaseFactory &dbFactory);
 	SaveFileResult saveFile(QFileInfo fileInfo, QVector<PageData> &pageData);
-	int getFiles(QVector<FileData> &results, QString wildCardPattern, int offset, int limit);
+	unsigned int getFiles(QVector<FileData> &results, QString wildCardPattern, int offset, int limit);
 	bool deleteFile(QString path);
 	bool fileExistsInDatabase(QString path);
 	bool fileExistsInDatabase(QString path, qint64 mtime);

shared/sqlitesearch.cpp

@@ -183,6 +183,11 @@ QSqlQuery SqliteSearch::makeSqlQuery(const LooqsQuery &query)
 				  whereSql + " " + sortSql;
 	}
 
+	if(query.getLimit() > 0)
+	{
+		prepSql += " LIMIT " + QString::number(query.getLimit());
+	}
+
 	QSqlQuery dbquery(*db);
 	dbquery.prepare(prepSql);
 	for(const QString &value : bindValues)

shared/token.h

@@ -4,24 +4,25 @@
 
 enum TokenType
 {
-	WORD,
-	NEGATION = 2,
-	BOOL = 4,
+	WORD = 8,
+	NEGATION = 16,
+	BOOL = 32,
 	BOOL_AND,
 	BOOL_OR,
-	GROUP = 8,
+	GROUP = 64,
 	BRACKET_OPEN,
 	BRACKET_CLOSE,
-	SORT = 16,
-	FILTER_PATH = 32,
+	SORT = 128,
+	FILTER_PATH = 256,
 	FILTER_PATH_MTIME,
 	FILTER_PATH_CONTAINS,
 	FILTER_PATH_SIZE,
 	FILTER_PATH_ENDS,
 	FILTER_PATH_STARTS,
-	FILTER_CONTENT = 64,
+	FILTER_CONTENT = 512,
 	FILTER_CONTENT_CONTAINS,
 	FILTER_CONTENT_PAGE,
+	LIMIT = 1024
 };
 
 class Token

shared/wildcardmatcher.cpp

@@ -0,0 +1,29 @@
+#include "wildcardmatcher.h"
+
+void WildcardMatcher::setPatterns(QStringList patterns)
+{
+	this->regexes.clear();
+	for(QString &str : patterns)
+	{
+		QRegExp regexp;
+		regexp.setPattern(str);
+		regexp.setPatternSyntax(QRegExp::WildcardUnix);
+		this->regexes.append(regexp);
+	}
+}
+
+WildcardMatcher::WildcardMatcher()
+{
+}
+
+bool WildcardMatcher::match(QString haystack) const
+{
+	for(const QRegExp &regexp : this->regexes)
+	{
+		if(regexp.exactMatch(haystack))
+		{
+			return true;
+		}
+	}
+	return false;
+}

shared/wildcardmatcher.h

@@ -0,0 +1,17 @@
+#ifndef WILDCARDMATCHER_H
+#define WILDCARDMATCHER_H
+#include <QStringList>
+#include <QRegExp>
+class WildcardMatcher
+{
+  private:
+	QVector<QRegExp> regexes;
+	QStringList patterns;
+
+  public:
+	WildcardMatcher();
+	bool match(QString haystack) const;
+	void setPatterns(QStringList patterns);
+};
+
+#endif // WILDCARDMATCHER_H