Release v0.7

The current expanding setting resizes, somehow, the mainwindow
when a path name is quite long.

CHANGELOG.md

@@ -1,4 +1,20 @@
 # looqs: Release notes
+## 2022-09-10 - v0.7
+
+CHANGES:
+
+ - GUI: Add vertical scroll option, default to it. Most feedback considered horizontal scrolling unnatural.
+ - GUI: Previews: Improve plaintext preview snippet selection by prioritizing those which contain the most search terms
+ - GUI: Previews: Only highlight whole words, not parts in words.
+ - GUI: Previews: Don't treat text wrapped inside '<' '>' in plaintext files as HTMl tags which caused such text to get lost in previews.
+ - GUI: Avoid triggering preview generation in some cases even when previews tab is not the active one
+ - GUI: Implement a search history. Allow going up/down with arrow keys in the search field.
+ - GUI: Previews: Allow CTRL + mouse wheel to zoom in on previews
+ - General: Fix an incorrect sqlite query which caused the ranking information of search results to be lost. "All files" filter in the previews tab therefore now orders by the seemingly most relevant pages, across all documents.
+ - General: Fix handling of content search queries with prefix terms (those ending in '*').
+ - GUI: Show how many results are previewable.
+ - GUI: Refactor to improve stability of sandboxed IPC worker in order to avoid rare segfaults.
+
 ## 2022-08-14 - v0.6
 This release features multiple fixes and enhancements.
 

README.md

@@ -28,7 +28,7 @@ There is no need to write the long form of filters. There are also booleans avai
 The screenshots in this section may occasionally be slightly outdated, but they are usually recent enough to get an overall impression of the current state of the GUI.
 
 ## Current status
-Latest version: 2022-08-14, v0.6
+Latest version: 2022-09-10, v0.7
 
 Please see [Changelog](CHANGELOG.md) for a human readable list of changes.
 

USAGE.md

@@ -47,16 +47,23 @@ you can quickly perform content searches in paths containing 'docs'.
 
 **CTRL + W**: Removes the last filter. If we take above's example "p:(docs) c:(invoice credit card)" again, then CTRL + W kills "c:(invoice credit card)".
 
+The arrow keys (up and down) can be used to go back and forward in the search history.
+
 ### Configuring PDF viewer
 It's most convenient if, when you click on a preview, the PDF reader opens the page you clicked. For that, looqs needs to know which viewer you want to launch.
 
 It tries to auto detect some common viewers. You must set the value in the "Settings" tab yourself if the
 default does not work for you. In the command line options, "%f" represents the filepath, "%p" the page number.
 
-### Preview tab
+### Previews tab
-The preview tab shows previews. It marks your search keywords too. Click on a preview to open the file.
+The 'previews' tab shows previews. It marks your search keywords too. Click on a preview to open the file.
 A right click on a preview allows you to copy the file path, or to open the containing folder.
 
+When the combobox is set to "All previews", the previews are ordered by relevance from all documents/pages.
+
+By default, a vertical scrolling is active. In the settings, it can be changed to horizontal scroll, which may be
+preferred by users of (larger) wide screen monitors.
+
 ### Syncing index
 Over time, files get deleted or their content changes. Go to **looqs** -> **Sync index**. looqs will reindex the content of files which have been changed. Files that cannot be found anymore will be removed from the index.
 

gui/ipcpreviewworker.cpp

@@ -1,16 +1,50 @@
 #include <QtConcurrent>
 #include "ipcpreviewworker.h"
 #include "previewgeneratormapfunctor.h"
-IPCPreviewWorker::IPCPreviewWorker()
+IPCPreviewWorker::IPCPreviewWorker(QLocalSocket *peer)
 {
+	this->peer = peer;
 	this->connect(&previewWorkerWatcher, &QFutureWatcher<QByteArray>::resultReadyAt, this,
-				  [this](int index) { emit previewGenerated(previewWorkerWatcher.resultAt(index)); });
+				  [this](int index)
-	connect(&previewWorkerWatcher, &QFutureWatcher<QByteArray>::finished, this, [this] { emit finished(); });
+				  {
+					  if(this->peer != nullptr)
+					  {
+						  QDataStream stream{this->peer};
+						  stream << previewWorkerWatcher.resultAt(index);
+						  this->peer->flush();
+					  }
+				  });
+	connect(&previewWorkerWatcher, &QFutureWatcher<QByteArray>::finished, this, &IPCPreviewWorker::shutdownSocket);
+	connect(this->peer, &QLocalSocket::disconnected, this, &IPCPreviewWorker::shutdownSocket);
 }
 
-void IPCPreviewWorker::start(RenderConfig config, const QVector<RenderTarget> &targets, QLocalSocket *peer)
+void IPCPreviewWorker::shutdownSocket()
+{
+	if(cleaned)
+	{
+		return;
+	}
+	cleaned = true;
+	if(this->peer != nullptr)
+	{
+		if(this->peer->state() == QLocalSocket::ConnectedState)
+		{
+			this->peer->flush();
+			this->peer->waitForBytesWritten();
+			this->peer->disconnectFromServer();
+			if(this->peer->state() != QLocalSocket::UnconnectedState)
+			{
+				this->peer->waitForDisconnected();
+			}
+		}
+		delete this->peer;
+		this->peer = nullptr;
+	}
+	emit finished();
+}
+
+void IPCPreviewWorker::start(RenderConfig config, const QVector<RenderTarget> &targets)
 {
-	stop();
 	auto mapFunctor = PreviewGeneratorMapFunctor();
 	mapFunctor.setRenderConfig(config);
 

gui/ipcpreviewworker.h

@@ -11,13 +11,21 @@ class IPCPreviewWorker : public QObject
 	Q_OBJECT
   private:
 	QFutureWatcher<QByteArray> previewWorkerWatcher;
+	QLocalSocket *peer;
+	bool cleaned = false;
 
   public:
-	IPCPreviewWorker();
+	IPCPreviewWorker(QLocalSocket *peer);
-	void start(RenderConfig config, const QVector<RenderTarget> &targets, QLocalSocket *peer);
+	void start(RenderConfig config, const QVector<RenderTarget> &targets);
 	void stop();
+	~IPCPreviewWorker()
+	{
+		delete this->peer;
+	}
+  private slots:
+	void shutdownSocket();
+
   signals:
-	void previewGenerated(QByteArray);
 	void finished();
 };
 

gui/ipcserver.cpp

@@ -18,8 +18,6 @@ IpcServer::IpcServer()
 	/* Only 1, we are doing work for the GUI, not a service for general availability */
 	this->spawningServer.setMaxPendingConnections(1);
 	connect(&this->spawningServer, &QLocalServer::newConnection, this, &IpcServer::spawnerNewConnection);
-	connect(&this->previewWorker, &IPCPreviewWorker::previewGenerated, this, &IpcServer::handlePreviewGenerated);
-	connect(&this->previewWorker, &IPCPreviewWorker::finished, this, [this] { this->currentSocket->flush(); });
 }
 
 bool IpcServer::startSpawner(QString socketPath)
@@ -31,8 +29,6 @@ bool IpcServer::startSpawner(QString socketPath)
 void IpcServer::spawnerNewConnection()
 {
 	QLocalSocket *socket = this->spawningServer.nextPendingConnection();
-	connect(socket, &QLocalSocket::disconnected, socket, &QLocalSocket::deleteLater);
-	this->currentSocket = socket;
 	if(socket != nullptr)
 	{
 		if(!socket->waitForReadyRead())
@@ -53,21 +49,22 @@ void IpcServer::spawnerNewConnection()
 				stream.startTransaction();
 				stream >> renderConfig >> targets;
 			} while(!stream.commitTransaction() && socket->state() == QLocalSocket::ConnectedState);
-
+			if(socket->state() == QLocalSocket::ConnectedState)
-			stream << targets.count();
+			{
-			socket->flush();
+				stream << targets.count();
-			previewWorker.start(renderConfig, targets, socket);
+				socket->flush();
+				IPCPreviewWorker *previewWorker = new IPCPreviewWorker(socket);
+				connect(previewWorker, &IPCPreviewWorker::finished, this, [previewWorker] { delete previewWorker; });
+				previewWorker->start(renderConfig, targets);
+			}
+			else
+			{
+				delete socket;
+			}
 		}
 		if(command == StopGeneratePreviews)
 		{
-			previewWorker.stop();
+			/* TODO: implement */
 		}
 	}
 }
-
-void IpcServer::handlePreviewGenerated(QByteArray ba)
-{
-	QDataStream stream{this->currentSocket};
-	stream << ba;
-	this->currentSocket->flush();
-}

gui/ipcserver.h

@@ -10,13 +10,10 @@ class IpcServer : public QObject
 {
 	Q_OBJECT
   private:
-	IPCPreviewWorker previewWorker;
 	QLocalServer spawningServer;
-	QLocalSocket *currentSocket = nullptr;
 	SaveFileResult addFile(QString file);
   private slots:
 	void spawnerNewConnection();
-	void handlePreviewGenerated(QByteArray ba);
 
   public:
 	IpcServer();

gui/mainwindow.cpp

@@ -15,6 +15,7 @@
 #include <QFileDialog>
 #include <QScreen>
 #include <QProgressDialog>
+#include <QDesktopWidget>
 #include "mainwindow.h"
 #include "ui_mainwindow.h"
 #include "clicklabel.h"
@@ -647,6 +648,7 @@ void MainWindow::previewReceived(QSharedPointer<PreviewResult> preview, unsigned
 		headerLabel->setText(QString("Path: ") + preview->getDocumentPath());
 
 		ClickLabel *label = dynamic_cast<ClickLabel *>(preview->createPreviewWidget());
+		label->setMaximumWidth(QApplication::desktop()->availableGeometry().width() - 200);
 
 		QVBoxLayout *previewLayout = new QVBoxLayout();
 
@@ -678,6 +680,7 @@ void MainWindow::previewReceived(QSharedPointer<PreviewResult> preview, unsigned
 		previewLayout->setMargin(0);
 		previewLayout->insertStretch(0, 1);
 		previewLayout->insertStretch(-1, 1);
+		previewLayout->setAlignment(Qt::AlignCenter);
 		QWidget *previewWidget = new QWidget();
 
 		previewWidget->setLayout(previewLayout);
@@ -806,6 +809,7 @@ void MainWindow::handleSearchResults(const QVector<SearchResult> &results)
 	ui->comboPreviewFiles->clear();
 	ui->comboPreviewFiles->addItem("All previews");
 	ui->comboPreviewFiles->setVisible(true);
+	ui->lblTotalPreviewPagesCount->setText("");
 
 	bool hasDeleted = false;
 	QHash<QString, bool> seenMap;
@@ -816,7 +820,6 @@ void MainWindow::handleSearchResults(const QVector<SearchResult> &results)
 
 		if(!seenMap.contains(absPath))
 		{
-			seenMap[absPath] = true;
 			QString fileName = pathInfo.fileName();
 			QTreeWidgetItem *item = new QTreeWidgetItem(ui->treeResultsList);
 
@@ -830,17 +833,18 @@ void MainWindow::handleSearchResults(const QVector<SearchResult> &results)
 		bool exists = pathInfo.exists();
 		if(exists)
 		{
-			if(!result.wasContentSearch)
+			if(result.wasContentSearch)
 			{
-				continue;
+				if(!pathInfo.suffix().contains("htm")) // hack until we can preview them properly...
-			}
-
-			if(!pathInfo.suffix().contains("htm")) // hack until we can preview them properly...
-			{
-				if(PreviewGenerator::get(pathInfo) != nullptr)
 				{
-					this->previewableSearchResults.append(result);
+					if(PreviewGenerator::get(pathInfo) != nullptr)
-					ui->comboPreviewFiles->addItem(result.fileData.absPath);
+					{
+						this->previewableSearchResults.append(result);
+						if(!seenMap.contains(result.fileData.absPath))
+						{
+							ui->comboPreviewFiles->addItem(result.fileData.absPath);
+						}
+					}
 				}
 			}
 		}
@@ -848,6 +852,7 @@ void MainWindow::handleSearchResults(const QVector<SearchResult> &results)
 		{
 			hasDeleted = true;
 		}
+		seenMap[absPath] = true;
 	}
 
 	ui->treeResultsList->resizeColumnToContents(0);
@@ -863,6 +868,7 @@ void MainWindow::handleSearchResults(const QVector<SearchResult> &results)
 	}
 
 	QString statusText = "Results: " + QString::number(results.size()) + " files";
+	statusText += ", previewable: " + QString::number(this->previewableSearchResults.count());
 	if(hasDeleted)
 	{
 		statusText += " WARNING: Some files are inaccessible. No preview available for those. Index may be out of sync";

gui/mainwindow.ui

@@ -7,7 +7,7 @@
     <x>0</x>
     <y>0</y>
     <width>1280</width>
-    <height>888</height>
+    <height>923</height>
    </rect>
   </property>
   <property name="windowTitle">
@@ -27,7 +27,7 @@
        <enum>QTabWidget::South</enum>
       </property>
       <property name="currentIndex">
-       <number>3</number>
+       <number>1</number>
       </property>
       <widget class="QWidget" name="resultsTab">
        <attribute name="title">
@@ -82,7 +82,7 @@
              <x>0</x>
              <y>0</y>
              <width>1244</width>
-             <height>598</height>
+             <height>633</height>
             </rect>
            </property>
            <layout class="QHBoxLayout" name="horizontalLayout"/>
@@ -165,7 +165,7 @@
           <item>
            <widget class="QComboBox" name="comboPreviewFiles">
             <property name="sizePolicy">
-             <sizepolicy hsizetype="Expanding" vsizetype="Fixed">
+             <sizepolicy hsizetype="Maximum" vsizetype="Fixed">
               <horstretch>0</horstretch>
               <verstretch>0</verstretch>
              </sizepolicy>

gui/previewgeneratorpdf.cpp

@@ -1,5 +1,6 @@
 #include <QMutexLocker>
 #include <QPainter>
+#include <QRegularExpression>
 #include "previewgeneratorpdf.h"
 
 static QMutex cacheMutex;
@@ -17,6 +18,7 @@ Poppler::Document *PreviewGeneratorPdf::document(QString path)
 		return nullptr;
 	}
 	result->setRenderHint(Poppler::Document::TextAntialiasing);
+
 	QMutexLocker locker(&cacheMutex);
 	documentcache.insert(path, result);
 	locker.unlock();
@@ -45,7 +47,12 @@ QSharedPointer<PreviewResult> PreviewGeneratorPdf::generate(RenderConfig config,
 	QImage img = pdfPage->renderToImage(config.scaleX, config.scaleY);
 	for(QString &word : config.wordsToHighlight)
 	{
-		QList<QRectF> rects = pdfPage->search(word, Poppler::Page::SearchFlag::IgnoreCase);
+		QList<QRectF> rects =
+			pdfPage->search(word, Poppler::Page::SearchFlag::IgnoreCase | Poppler::Page::SearchFlag::WholeWords);
+		if(rects.empty())
+		{
+			rects = pdfPage->search(word, Poppler::Page::SearchFlag::IgnoreCase);
+		}
 		for(QRectF &rect : rects)
 		{
 			QPainter painter(&img);

gui/previewgeneratorplaintext.cpp

@@ -1,4 +1,5 @@
 #include <QTextStream>
+#include <QRegularExpression>
 
 #include "previewgeneratorplaintext.h"
 #include "previewresultplaintext.h"
@@ -57,6 +58,7 @@ QString PreviewGeneratorPlainText::generatePreviewText(QString content, RenderCo
 		++i;
 	}
 
+	resulText = resulText.toHtmlEscaped();
 	QString header = "<b>" + fileName + "</b> ";
 	for(QString &word : config.wordsToHighlight)
 	{
@@ -74,10 +76,19 @@ QString PreviewGeneratorPlainText::generatePreviewText(QString content, RenderCo
 	return header + resulText.replace("\n", "<br>").mid(0, 1000);
 }
 
+struct Snippet
+{
+	/* Contains each line number and line of the snippet*/
+	QString snippetText;
+
+	/* How many times a word occurs in the snippetText */
+	QHash<QString, int> wordCountMap;
+};
+
 QString PreviewGeneratorPlainText::generateLineBasedPreviewText(QTextStream &in, RenderConfig config, QString fileName)
 {
-	QString resultText;
+	QVector<Snippet> snippets;
-	const unsigned int contextLinesCount = 2;
+	const int contextLinesCount = 2;
 	LimitQueue<QString> queue(contextLinesCount);
 	QString currentLine;
 	currentLine.reserve(512);
@@ -85,38 +96,73 @@ QString PreviewGeneratorPlainText::generateLineBasedPreviewText(QTextStream &in,
 	/* How many lines to read after a line with a match (like grep -A ) */
 	int justReadLinesCount = -1;
 
-	auto appendLine = [&resultText](int lineNumber, QString &line)
+	struct Snippet currentSnippet;
-	{ resultText.append(QString("<b>%1</b>%2<br>").arg(lineNumber).arg(line)); };
 
-	QHash<QString, int> countmap;
+	auto appendLine = [&currentSnippet, &config](int lineNumber, QString &line)
-	QString header = "<b>" + fileName + "</b> ";
-
-	unsigned int snippetsCount = 0;
-	unsigned int lineCount = 0;
-	while(in.readLineInto(&currentLine) && snippetsCount < MAX_SNIPPETS)
 	{
+		int foundWordsCount = 0;
+		for(QString &word : config.wordsToHighlight)
+		{
+			QRegularExpression searchRegex("\\b" + word + "\\b");
+			bool containsRegex = line.contains(searchRegex);
+			bool contains = false;
+			if(!containsRegex)
+			{
+				contains = line.contains(word, Qt::CaseInsensitive);
+			}
+			if(containsRegex || contains)
+			{
+				currentSnippet.wordCountMap[word] = currentSnippet.wordCountMap.value(word, 0) + 1;
+				QString replacementString = "<span style=\"background-color: yellow;\">" + word + "</span>";
+				if(containsRegex)
+				{
+					line.replace(searchRegex, replacementString);
+				}
+				else
+				{
+					line.replace(word, replacementString, Qt::CaseInsensitive);
+				}
+				++foundWordsCount;
+			}
+		}
+		currentSnippet.snippetText.append(QString("<b>%1</b>%2<br>").arg(lineNumber).arg(line));
+		return foundWordsCount;
+	};
+
+	unsigned int lineCount = 0;
+	while(in.readLineInto(&currentLine))
+	{
+		currentLine = currentLine.toHtmlEscaped();
 		++lineCount;
 		bool matched = false;
 		if(justReadLinesCount > 0)
 		{
-			appendLine(lineCount, currentLine);
+
-			--justReadLinesCount;
+			int result = appendLine(lineCount, currentLine);
+			if(justReadLinesCount == 1 && result > 0)
+			{
+				justReadLinesCount = contextLinesCount;
+			}
+			else
+			{
+				--justReadLinesCount;
+			}
+
 			continue;
 		}
 		if(justReadLinesCount == 0)
 		{
-			resultText += "---<br>";
+			currentSnippet.snippetText += "---<br>";
 			justReadLinesCount = -1;
-			++snippetsCount;
+			snippets.append(currentSnippet);
+			currentSnippet = {};
 		}
 		for(QString &word : config.wordsToHighlight)
 		{
 			if(currentLine.contains(word, Qt::CaseInsensitive))
 			{
-				countmap[word] = countmap.value(word, 0) + 1;
 				matched = true;
-				currentLine.replace(word, "<span style=\"background-color: yellow;\">" + word + "</span>",
+				break;
-									Qt::CaseInsensitive);
 			}
 		}
 		if(matched)
@@ -125,7 +171,6 @@ QString PreviewGeneratorPlainText::generateLineBasedPreviewText(QTextStream &in,
 			{
 				int queuedLineCount = lineCount - queue.size();
 				QString queuedLine = queue.dequeue();
-
 				appendLine(queuedLineCount, queuedLine);
 			}
 			appendLine(lineCount, currentLine);
@@ -137,13 +182,77 @@ QString PreviewGeneratorPlainText::generateLineBasedPreviewText(QTextStream &in,
 		}
 	}
 
+	if(!currentSnippet.snippetText.isEmpty())
+	{
+		currentSnippet.snippetText += "---<br>";
+		snippets.append(currentSnippet);
+	}
+
+	std::sort(snippets.begin(), snippets.end(),
+			  [](Snippet &a, Snippet &b)
+			  {
+				  int differentWordsA = 0;
+				  int totalWordsA = 0;
+				  int differentWordsB = 0;
+				  int totalWordsB = 0;
+				  for(int count : a.wordCountMap.values())
+				  {
+					  if(count > 0)
+					  {
+						  ++differentWordsA;
+					  }
+					  totalWordsA += count;
+				  }
+				  for(int count : b.wordCountMap.values())
+				  {
+					  if(count > 0)
+					  {
+						  ++differentWordsB;
+					  }
+					  totalWordsB += count;
+				  }
+
+				  if(differentWordsA > differentWordsB)
+				  {
+					  return true;
+				  }
+				  if(differentWordsA == differentWordsB)
+				  {
+					  return totalWordsA > totalWordsB;
+				  }
+				  return false;
+			  });
+
+	QString resultText = "";
+
+	unsigned int snippetsCount = 0;
+
+	QString header = "<b>" + fileName + "</b> ";
+
+	QHash<QString, int> totalWordCountMap;
+	bool isTruncated = false;
+	for(Snippet &snippet : snippets)
+	{
+		if(snippetsCount++ < MAX_SNIPPETS)
+		{
+			resultText += snippet.snippetText;
+		}
+		else
+		{
+			isTruncated = true;
+		}
+		for(auto it = snippet.wordCountMap.keyValueBegin(); it != snippet.wordCountMap.keyValueEnd(); it++)
+		{
+			totalWordCountMap[it->first] = totalWordCountMap.value(it->first, 0) + it->second;
+		}
+	}
+	if(isTruncated)
+	{
+		header += "(truncated) ";
+	}
 	for(QString &word : config.wordsToHighlight)
 	{
-		header += word + ": " + QString::number(countmap[word]) + " ";
+		header += word + ": " + QString::number(totalWordCountMap[word]) + " ";
-	}
-	if(snippetsCount == MAX_SNIPPETS)
-	{
-		header += "(truncated)";
 	}
 	header += "<hr>";
 

shared/limitqueue.h

@@ -6,11 +6,11 @@ template <class T> class LimitQueue
 {
   protected:
 	QQueue<T> queue;
-	unsigned int limit = 0;
+	int limit = 0;
 
   public:
 	LimitQueue();
-	LimitQueue(unsigned int limit)
+	LimitQueue(int limit)
 	{
 		this->limit = limit;
 	}
@@ -34,7 +34,7 @@ template <class T> class LimitQueue
 		return queue.dequeue();
 	}
 
-	void setLimit(unsigned int limit)
+	void setLimit(int limit)
 	{
 		this->limit = limit;
 	}